How to tag and classify sensitive data (PII) automatically

Summary

• Databricks Unity Catalog uses AI-powered tagging to automatically detect and classify personally identifiable information (PII) across your data estate, scanning table schemas and content patterns to identify sensitive columns.
• Once columns are tagged with governed tags like PII.Email or PII.Phone, you can enforce policies uniformly across all workloads: column masks hide values from unauthorized users, row filters restrict access, and queries are blocked if appropriate permissions are missing.
• The classification workflow runs on-demand or scheduled, integrating directly into Unity Catalog's access control layer so policies take effect immediately without manual schema changes.
• Governed tags link to RBAC and ABAC policies, which are enforced at query time across SQL, notebooks, jobs, BI tools, and Genie, ensuring one consistent rule set regardless of how data is accessed.

How to tag and classify sensitive data (PII) automatically

Automated PII detection identifies sensitive columns without manual inspection, tagging them so access policies apply consistently. Unity Catalog's governed tags create a common vocabulary: once a column is tagged as PII, you attach policies that mask values, restrict rows, or deny queries (all enforced at query time so the same rule applies whether data is queried from SQL, a notebook, a BI tool, or an AI agent).

What is PII classification?

PII classification identifies columns containing personally identifiable information (email, phone numbers, social security numbers, credit card data) and labels them with standardized tags. This enables consistent governance: once tagged, a column can be masked, filtered, or restricted without modifying the table schema.

Key capabilities to evaluate

Capability	Why it matters
Pattern-based detection	Scans column names and data types for common PII patterns (e.g., email, ssn, credit_card_number) without manual rule authoring.
Content scanning	Analyzes data values to identify PII, not just metadata, catching sensitive information even when column names are generic.
Custom tag definitions	Teams define sensitivity levels and classifications specific to their business (e.g., Internal.Confidential, Compliance.GDPR).
Bulk application and workflow	Tags multiple columns at once and integrates with scheduled jobs so new data is classified continuously.
Governed tag enforcement	Ties tags directly to policies so tagging decisions propagate to column masks, row filters, and permission checks.
Audit trail	Records who tagged what and when, enabling compliance reporting.

How Databricks approaches PII classification

Unity Catalog provides the classification infrastructure using governed tags (metadata labels that are centrally defined and enforced). Once a column is tagged, either manually or via automated detection, you create policies that reference those tags:

• Automated tagging at scale: Use Unity Catalog APIs to scan and tag columns programmatically. Pattern-matching identifies common PII formats, and SQL queries on information schema let you validate migrations and find untagged sensitive columns.
• Column masking with tags: Once a column carries a PII tag, define a mask that hides values from all users except those explicitly granted permission. Column masks apply at query time across all query surfaces (SQL, notebooks, Python, BI tools).
• Row-level filtering: Row filters tied to PII tags restrict which rows a user can see based on groups or attributes, useful for multi-tenant data or regional compliance rules.
• Attribute-based access control (ABAC): Use ABAC policies to grant or deny permissions based on tags rather than object-by-object grants. Example: a policy can state "Users in Finance can query tables tagged Internal.Financial, but not Compliance.Confidential." Tagging decisions flow through automatically.
• Scheduled classification jobs: Automate PII detection by running classification logic on a schedule so new tables and columns are tagged as data arrives. Use Lakeflow Jobs or Lakeflow Pipelines to orchestrate.
• Compliance and audit: Unity Catalog logs all tagging events, providing a complete record for regulatory reporting.

See how to scale data governance with ABAC for a hands-on walkthrough.

Use cases

• Financial services: A bank runs automatic PII detection on customer deposit, loan, and transaction tables, tagging columns with PII.AccountHolder and Compliance.GLBA. Row filters ensure each branch sees only their customer data, and column masks hide SSNs unless explicitly approved.
• Healthcare: A healthcare provider tags patient records with PII.Medical. ABAC policies allow clinicians to query records but automatically mask medical record numbers from billing staff. Regular automated scans catch new datasets on arrival.
• Retail: A retailer tags customer email, phone, and address columns as PII.Contact, then uses column masks to show anonymized versions to analytics teams while giving customer service representatives unmasked access.

FAQs

How do I start classifying PII in my existing data?

Begin with a manual audit of high-risk tables (customer, order, user, payment) and tag key columns using Unity Catalog APIs. Create column masks and row filters tied to those tags. Capture the classification logic in a reusable job so new datasets are tagged automatically.

Can I define custom sensitivity categories beyond PII?

Yes. Governed tags are fully customizable. Create hierarchies like Internal.Public, Internal.Confidential, and Restricted.Regulatory tailored to your governance framework.

What happens if a user tries to query a masked PII column without permission?

The mask is applied at query time: authorized users see the real value, unauthorized users see a redacted version (e.g., XXX-XX-1234 for a masked SSN). The query succeeds but returns transformed data. To block the query entirely, use a row filter or privilege that denies SELECT permission.

How often should I re-run automated classification?

Run it weekly or daily, depending on your data ingestion cadence. Use Lakeflow Jobs to schedule automatically so new tables and columns are tagged as soon as data lands.

Can I enforce PII policies across BI tools and Genie?

Yes. All policies defined in Unity Catalog (column masks, row filters, privileges) are enforced at the SQL engine level, applying uniformly across SQL, notebooks, BI dashboards, and AI agents like Genie. No policy duplication needed.