how to store petabytes of images and video for AI training

Summary

• Keep images and video as files in cloud object storage and govern them as Unity Catalog volumes. Volumes enable governance over non-tabular data of any format, and you can't register their files as tables.
• Keep a Delta table as the queryable index. read_files with the binaryFile format returns the path, modification time, length and content of each file, and you can exclude content so the index holds metadata only.
• Set access at the volume level. READ VOLUME and WRITE VOLUME are granted on a volume (or on its catalog or schema), and the docs describe no per-file grants, so put data with different access needs in separate volumes.
• For training, avoid millions of tiny reads. The AI Runtime data loading guide (Public Preview) recommends files of roughly 1 GiB to 10 GiB, and Mosaic Streaming can convert a Spark DataFrame and save it to a volume for single-node or distributed training.
• Log the Delta table version with each training run so Unity Catalog can link model versions to the data they trained on.

how to store petabytes of images and video for AI training

Store the files in cloud object storage as a Unity Catalog volume, keep labels and splits in a Delta table that indexes them, and load training data in large files instead of millions of small ones. The volume is the governed file layer. The Delta table is the queryable catalog of what is in it.

What does each layer do?

Layer	Holds	Databricks feature
Files	Raw images and video	Unity Catalog volume, managed or external, on cloud object storage
Index	Path, size, modification time, labels, splits	Delta table in Unity Catalog, built with read_files
Ingestion	New files as they arrive	Auto Loader, which supports the BINARYFILE format
Training input	Files read in large batches	/Volumes/... paths, Mosaic Streaming, UCVolumeDataset (Public Preview)
Governance	Who can read, and what trained on what	Volume privileges, row filters and column masks on tables, tags, lineage, audit logs

How do I build the index table?

Volumes give you paths, not tables: "You can't register files in volumes as tables in Unity Catalog." This query, adapted from the read_files docs, reads unstructured files without their bytes and saves the result as a table:

CREATE TABLE my_catalog.my_schema.image_index AS
SELECT * EXCEPT (content), _metadata
FROM read_files(
  '/Volumes/my_catalog/my_schema/my_volume',
  format => 'binaryFile');

The binaryFile source returns path, modificationTime, length and content. The hidden _metadata column adds file_path, file_name, file_size and file_modification_time. Join your own label, split and source columns onto this table. For files that keep arriving, Auto Loader "incrementally and efficiently processes new data files as they arrive in cloud storage" and supports BINARYFILE.

How do I load petabyte-scale data for training?

Volumes provide POSIX-style paths, which Databricks calls ideal for "machine learning frameworks and open source Python modules that require POSIX-style access." For petabyte-scale image and video sets, file size matters. The AI Runtime data loading guide (Public Preview) says to "Consolidate data into fewer, larger files, roughly 1 GiB to 10 GiB per file." That lets AI Runtime "aggressively prefetch data and automatically achieve near-optimal sequential read performance." For many small files, it suggests "copying the files to the local disk using parallel copies before processing them." Two documented loaders help:

• Mosaic Streaming is compatible with "any data type, including images, text, video, and multimodal data." dataframe_to_mds can save a DataFrame to a Unity Catalog volume, and StreamingDataset has "elastically deterministic shuffling, which enables fast mid-epoch resumption." It is pre-installed in Databricks Runtime 15.2 ML and higher.
• UCVolumeDataset (AI Runtime, Public Preview) is a PyTorch IterableDataset that copies each file from the volume to a local cache on first access. When torch.distributed is initialized, it partitions files across ranks.

How do I govern and audit it?

• Access. READ VOLUME covers reading or listing files, and WRITE VOLUME covers creating, deleting or updating them. Set them on a volume, or on a catalog or schema so they cascade. Split regions or sensitive sources into separate volumes.
• Index table. Row filters and column masks apply to tables, not views, and the docs state: "Time travel isn't supported with table-level row filters or column masks." Keep the versioned training set in a table without those policies.
• Tags. Tags organize and categorize securable objects, including volumes, tables and columns.
• Lineage. Unity Catalog captures lineage down to the column level, and lineage graphs include file paths and ML model versions.
• Audit. The audit log reference documents generateTemporaryVolumeCredential ("A temporary credential is granted for a volume") and generateTemporaryTableCredential, which you can use "to determine who queried what and when". The audit log system table is system.access.audit.

FAQs

How do I version training data across experiments?

Delta time travel reads a table with VERSION AS OF or TIMESTAMP AS OF, within retention limits: 30 days of log history and 7 days of deleted data files by default. Log the version with the run:

dataset = mlflow.data.load_delta(table_name="prod.ml_team.image_index", version="0")
with mlflow.start_run():
    mlflow.log_input(dataset, "training")

Table-to-model lineage needs MLflow 2.11.0 or above.

Managed or external volume?

Databricks recommends external volumes for files that external systems also read or write. For workloads that run only on Databricks, managed volumes are the simplest way to create governed storage. With external volumes, Unity Catalog doesn't govern reads and writes made directly against cloud object storage from external systems, so you must apply additional governance outside Databricks.

Can I upload a large video in the UI?

The UI upload limit is 5 GB per file. For larger files, use the Databricks SDK for Python.

Can I relabel images without rewriting files?

Yes. Labels live in the Delta table, and Delta Lake gives table updates ACID guarantees.