What is data governance?

Summary

• Data governance is a comprehensive approach that comprises the principles, practices and tools to manage an organization's data assets throughout their lifecycle.
• Databricks documentation defines data and AI governance as the management of the availability, usability, integrity, and security of an organization's data and AI assets.
• Data security is one element of governance, alongside cataloging, quality, classification, auditing, lineage, discovery, and sharing.
• Unity Catalog is the unified governance layer for data and AI in Databricks.

What is data governance?

Data governance is a comprehensive approach that comprises the principles, practices and tools to manage an organization's data assets throughout their lifecycle. Databricks documentation describes data and AI governance as "the management of the availability, usability, integrity, and security of an organization's data and AI assets."

What are the key elements of data governance?

Databricks describes these key elements:

• Data cataloging: a centralized metadata repository for an organization's data assets.
• Data quality: accuracy, completeness, freshness, and compliance with data-quality rules.
• Data classification: organizing and categorizing data by its sensitivity, value, and criticality.
• Data security: access controls that define which groups or individuals can access what data.
• Auditing data entitlements and access: understanding who has access to what data and tracking recent access.
• Data lineage: an end-to-end view of how data flows across an organization's data estate.
• Data discovery: making data easy to find for analytics, AI, or ML use cases.
• Data sharing and collaboration: exchanging data with internal teams, partners, and customers while keeping control and visibility over how it is used.

Who is responsible for data governance?

The Chief Data Officer is the most senior executive on the governance team and is ultimately responsible for the security, accessibility, and usability of the organization's data. Data owners are individuals or teams responsible for the technical administration of data sets. Data stewards implement the program, make sure both old and new data is managed appropriately, and monitor compliance. A data governance committee is the main body that creates policies.
In Unity Catalog, every securable object has an owner. Owners have all privileges on the object, including the ability to grant privileges to other principals.

Why does data governance matter?

Databricks lists these benefits: a single source of truth that prevents data sprawl and silos, accurate and trustworthy data, collaboration across teams, business units, and partners, controls that prevent unauthorized access to sensitive data, and better compliance with regulatory requirements such as HIPAA, FedRAMP, GDPR, or CCPA.

How does data governance relate to data security?

Databricks lists data security as one of the key elements of data governance and says governance "mitigates security and privacy risks by implementing controls and processes to prevent unauthorized access and misuse of sensitive data."
In the Databricks documentation, Unity Catalog governance capabilities include access control, governed tags, discovery, lineage, classification, quality monitoring, and auditing. Related security topics, such as authentication, networking, encryption, and privacy compliance, are covered in Security and compliance.

How do you implement data governance?

The Databricks guide to best practices for data and AI governance groups its recommendations under three principles:

1. Unify data and AI management: establish a governance process, design Unity Catalog for your organization, manage metadata for all data and AI assets in one place, track data and AI lineage, add consistent descriptions to metadata, allow easy data discovery, and govern AI assets together with data.
2. Unify data and AI security: centralize access control for all data and AI assets, configure audit logging, and audit data platform events.
3. Establish data quality standards: define clear data quality standards, use data quality tools for profiling, cleansing, validating, and monitoring data, and implement and enforce standardized data formats and definitions.

How does Unity Catalog support data governance?

Unity Catalog governs models, functions, and other AI assets as securable objects, the same way it governs tables and volumes.

Governance need	What Unity Catalog provides
Ownership	Every securable object has an owner with all privileges on it
Access control	Privileges, attribute-based access control, and row filters and column masks
Discovery	Catalog Explorer, AI-generated comments, and certified and deprecated tags
Lineage	Captured automatically for queries run on Databricks, down to the column level, across all workspaces attached to the metastore
Tags and classification	Governed tags that control who can assign tags, and automatic scanning and tagging of sensitive data such as PII across tables
Data quality	Anomaly detection for table freshness and completeness, and data profiling for summary statistics
Auditing	Audit logs of who accessed data and what actions they took
Retention	Auto time-to-live deletes rows from managed tables after a configurable period, and exact deletion timing is not guaranteed

FAQs

Is data governance the same as a data dictionary?

No. The IBM Dictionary of Computing defines a data dictionary, or metadata repository, as a centralized repository of information about data such as meaning, relationships to other data, origin, usage, and format. That corresponds to the data cataloging element of governance. Governance also covers data quality, classification, security, auditing, lineage, discovery, and sharing.

Is data governance only for regulated industries?

No. Compliance is one benefit among several. Databricks also lists a single source of truth that prevents data sprawl and silos, accurate and trustworthy data, and easier collaboration.

Where should a governance program start?

The Databricks list of principal steps begins with identifying all sources of data, preparing metadata and organizing a metadata storage option, and setting up mechanisms to track data lineage.

Does data governance cover AI assets?

Yes. Databricks frames the practice as data and AI governance, and Unity Catalog lineage includes ML model versions as nodes in the lineage graph.