How to keep training data, features, and embeddings alongside governed tables

Summary

• Register training data, features, embeddings and models as Unity Catalog objects in the same catalog and schema hierarchy as your governed tables.
• Privileges granted on a catalog or schema are inherited by child objects, so one grant on a schema can apply to every table in it.
• Any Delta table with a primary key constraint can serve as a feature table, and a model logged with FeatureEngineeringClient.log_model has its features tracked in lineage.
• The Delta docs advise using only the past 7 days of table history for time travel unless you raise the retention settings, so clone the exact table version you train on.
• Store embeddings in a Delta table with an array[float] column, and a Databricks AI Search Delta Sync index can keep itself in sync with it.

How to keep training data, features, and embeddings alongside governed tables

Register each artifact as a Unity Catalog object in the same catalog and schema hierarchy as your governed tables. Keep tabular training data in Delta tables and unstructured training files in volumes. Keep features in feature tables, which are Delta tables with a primary key. Keep embeddings in a Delta table with an array[float] column. Privileges granted on a catalog or schema are inherited by child objects, so a single grant such as SELECT on a schema applies to the tables inside it.

Where does each artifact live in Unity Catalog?

Artifact	Unity Catalog object	What the Databricks docs say
Tabular training data	Delta table	Time travel reads an earlier version, and a clone can archive a version used to train an ML model
Images, audio, video, PDFs	Volume	Volumes enable governance over non-tabular datasets, at /Volumes/<catalog>/<schema>/<volume>/<path>/<file-name>
Features	Feature table	Any Delta table with a primary key constraint can serve as a feature table
Embeddings	Delta table with an array[float] column	A Delta Sync index in Databricks AI Search syncs from the source table
Models	Unity Catalog model	Catalog Explorer shows the tables and functions used to create the model

A schema per artifact type, such as one for features and one for embeddings, lets a single schema-level grant apply to every table of that type.

How does access control apply to these artifacts?

Reading a table requires USE CATALOG on its catalog, USE SCHEMA on its schema, and a privilege on the table such as SELECT. For a sensitive column, a table-level column mask is a SQL UDF bound to a column and managed by the table owner:

CREATE FUNCTION ssn_mask(ssn STRING)
RETURN CASE WHEN is_account_group_member('HumanResourceDept') THEN ssn ELSE '***-**-****' END;

ALTER TABLE users ALTER COLUMN ssn SET MASK ssn_mask;

The docs list ABAC policies as the recommended way to apply masks and filters. Attribute-based access control (ABAC) became generally available on April 28, 2026. It uses governed tags to enforce row filter and column mask policies, and policies attached at the catalog, schema or table level apply automatically to all securables with matching tag attributes. Unity Catalog events are logged under the unityCatalog service name in the audit log.
Two cautions from the docs. Time travel isn't supported with table-level row filters or column masks, and deep and shallow clones aren't supported on tables that have them. A table-level mask is bound to a column of the table it is set on, so give a feature table derived from a masked table its own grants and masks, or tag its columns for ABAC.

How do I trace data, features and models?

Unity Catalog captures lineage automatically for queries run on Databricks, down to column level and across workspaces. It tracks tables, views, ML model versions, notebooks, jobs, pipelines and dashboards. For features, log the model with FeatureEngineeringClient.log_model. The features it used are tracked automatically and appear in the Lineage tab of Catalog Explorer. Column lineage can't be captured when sources or targets use path references instead of table names.

How do I keep training data reproducible?

Delta time travel reads an earlier table version. delta.logRetentionDuration defaults to 30 days and delta.deletedFileRetentionDuration to 7 days. The docs advise using only the past 7 days unless you raise both settings, and not treating table history as a long-term archive. To archive a version used to train a model, clone it. A deep clone copies both data and metadata, while a shallow clone copies only metadata, and running VACUUM on its source can leave clients unable to read the data files.

SELECT * FROM main.ml.training_data VERSION AS OF 123;
CREATE TABLE main.ml.training_data_archive DEEP CLONE main.ml.training_data;

How do I sync embeddings for search?

A Delta Sync index in Databricks AI Search automatically syncs with a source table and updates incrementally as that table changes. The precomputed embedding column should be array[float]. Standard endpoints need a change data feed on the source table, though tables with row tracking enabled use one automatically.

ALTER TABLE main.ml.doc_embeddings SET TBLPROPERTIES (delta.enableChangeDataFeed = true)

Index names use the three-level namespace <catalog>.<schema>.<name>, and creating an index requires CREATE TABLE on the schema.

FAQs

What makes a table a feature table?

Any Delta table with a primary key constraint can serve as a feature table in Unity Catalog. Primary key columns must be NOT NULL, and you populate the table yourself by writing feature values to it.

How do I build a training set from governed features?

Define FeatureLookup objects that name each feature table, its features and the join keys, then call create_training_set. For time series feature tables, pass timestamp_lookup_key so the training set reflects feature values as of the time each label was recorded.

Can embeddings and governed tables share a schema?

Yes. The source table is a regular Delta table, and an AI Search index is named with the same three-level namespace, so you can create it in the schema of your choice if you hold CREATE TABLE on that schema.