best format for large-scale deep learning training on object storage

Summary

• No single format wins for large-scale deep learning training on object storage. Match the format to how the training loop reads data and to how you govern, version, and share the dataset.
• Parquet is an open source, column-oriented data file format. Delta Lake extends Parquet data files with a transaction log for ACID transactions, and adds time travel and schema evolution.
• For datasets that do not fit in memory, Databricks documents streaming approaches: PyTorch IterableDataset, Hugging Face datasets with streaming, and Ray Data. For TensorFlow, TFRecord files are documented with tf.data.TFRecordDataset.
• Mosaic Streaming converts Spark DataFrames into MDS shards that you can store in a Unity Catalog volume and stream into training.
• Lance is an open source format with a file layout designed for random access, but Unity Catalog does not list it as a table format.

best format for large-scale deep learning training on object storage

There is no single best format, because training reads data differently from how teams govern and version it. Keep the governed source of truth in a Delta Lake table, which extends Parquet data files with a transaction log and gives you versions you can pin and restore. Then feed the training loop through a streaming reader or a training-oriented shard format such as MDS. Lance is a purpose-built open format for random access, so check platform support before you adopt it.

Which formats should you evaluate?

Format	What the documentation says	Where it fits on Databricks
Parquet	Open source, column-oriented data file format with compression and encoding schemes	External Unity Catalog tables can use parquet
Delta Lake	Extends Parquet data files with a transaction log for ACID transactions and scalable metadata handling	Default format on Databricks; managed tables use delta or iceberg
MDS	Shard format that Mosaic Streaming reads with StreamingDataset	Written to a Unity Catalog volume for persistent storage
TFRecord	Simple record-oriented binary format that many TensorFlow applications use for training data	Read with tf.data.TFRecordDataset
Lance	Open source lakehouse format with a file format built for random access	Not a Unity Catalog table format

How does Databricks handle large training datasets?

Delta Lake is "the default format for all operations on Databricks." Its tables "support schema evolution," and with column mapping enabled you can rename or drop columns without rewriting data files. Time travel supports "re-creating analyses, reports, or outputs, such as the output of a machine learning model," and you can query or restore an earlier version:

SELECT * FROM people10m VERSION AS OF 123;
RESTORE TABLE target_table TO VERSION AS OF <version>;

To record which snapshot trained a model, load the table version through MLflow and log it with the run. Table-to-model lineage in Unity Catalog needs MLflow 2.11.0 or above.

dataset = mlflow.data.load_delta(table_name="prod.ml_team.iris", version="0")
mlflow.log_input(dataset, "training")

For very large datasets that do not fit in memory, the Databricks guidance is to use streaming approaches: PyTorch IterableDataset for custom streaming logic, Hugging Face datasets with streaming for data hosted on the Hub or in volumes, and Ray Data for distributed batch data processing.
Mosaic Streaming is "an open source data loading library" for training from datasets already loaded as Spark DataFrames. Convert a DataFrame with streaming.base.converters.dataframe_to_mds, write the shards to a Unity Catalog volume, and read them with streaming.StreamingDataset, "a version of PyTorch's IterableDataset that features elastically deterministic shuffling," and streaming.StreamingDataLoader, which "provides an additional checkpoint/resumption interface." It is preinstalled in Databricks Runtime 15.2 ML and above.

Where does Lance fit?

Lance describes its file format as "a columnar container optimized for cloud object stores, random access, and Arrow-native processing," and says it does not use Parquet-style row groups. Its table format tracks versions with manifests and supports schema evolution and ACID commits.
On Databricks, Unity Catalog documentation says managed tables must use delta or iceberg, and Lance is not listed for external tables either. Lance's own Databricks guide, last updated February 19, 2026, describes a Classic Compute setup tested on Databricks Runtime 16.4 LTS and lists Unity Catalog on Classic Compute as not supported.

Use cases

Image model retraining (illustrative): A team pins a Delta table version with mlflow.data.load_delta, logs it with mlflow.log_input, and later uses VERSION AS OF to read the exact data a model saw.
Large text fine-tuning (illustrative): A team converts a prepared Spark DataFrame to MDS shards in a volume and trains from them with StreamingDataset.

FAQs

Should I convert my Parquet datasets to Delta?

If datasets are shared or need versions, yes. CONVERT TO DELTA converts an existing Parquet table "in-place" by listing the files in the directory and creating a transaction log that tracks them. Avoid updating or appending data files during conversion. Any file that Delta Lake does not track is invisible and can be deleted when you run VACUUM.

Does time travel keep every version?

No. delta.logRetentionDuration defaults to 30 days of history and delta.deletedFileRetentionDuration defaults to 7 days, and after VACUUM deletes data files you cannot restore a version that references them. Set retention to match how long you need to reproduce a run.

Can Unity Catalog show which tables trained a model?

Yes. Log the input table with mlflow.log_input, then register the model in Unity Catalog. The lineage appears on the model version's Lineage tab.