What's the best file format for a Spark data lake?

Summary

• Apache Parquet is the best default file format for a Spark data lake: it's Spark's default data source, it's columnar, it preserves the schema, and Spark turns on Parquet filter pushdown and vectorized decoding by default.
• Files alone don't make reliable tables. For data you update, delete, merge or read while it's being written, layer Delta Lake or Apache Iceberg on top; both keep the data in Parquet files by default and add a transaction log or metadata layer.
• Apache ORC is the other columnar format built into Spark. Spark's docs highlight its native zstd compression, bloom filters and columnar encryption.
• Avro is a serialization format whose files carry their own schema, which suits ingestion and record exchange; in Spark it's the external spark-avro module.
• Keep CSV and JSON at the landing edge: they have no embedded schema or column statistics, and CSV schema inference costs an extra pass over the data.

What's the best file format for a Spark data lake?

Store a Spark data lake in Apache Parquet. It's the format Spark reads and writes when you don't name one, it's columnar, and Spark's Parquet reader pushes filters down and decodes column batches without extra configuration. For tables that change over time, don't stop at files: put Delta Lake or Apache Iceberg over the Parquet files to get transactions, schema evolution and time travel.

File format or table format?

A file format sets how bytes are laid out inside one file. A table format records which data files belong to a table, version by version. Delta Lake writes its data as Parquet files and tracks each commit, and so each table version, in a transaction log. Apache Iceberg is "an open table format for huge analytic datasets," and its default data file format (write.format.default) is Parquet, with ORC and Avro as options. Picking a table format doesn't replace the Parquet decision; it builds on it.

How the formats compare in Spark

Format	What it is	Spark support	Use it for
Parquet	Columnar files	Built in and the default data source; filter pushdown and vectorized reader on by default; snappy compression by default	Default storage for analytical data
ORC	Columnar files	Built in (native implementation by default); filter pushdown and vectorized reader on by default; zstd compression by default	Lakes already standardized on ORC
Avro	Serialized records with the schema stored in the file	Built-in support through the external spark-avro module	Ingestion and record exchange between systems
CSV, JSON	Text	Built in; CSV schema inference needs one extra pass over the data	Raw landing data from other systems
Delta Lake, Apache Iceberg	Table formats over data files	Separate open source projects with Spark connectors	Tables you update, merge, version or stream into

Spark settings that shape a Parquet lake

• Compression. spark.sql.parquet.compression.codec defaults to snappy, and accepted values include zstd, gzip and lz4. Spark's ORC writer defaults to zstd, and so does Iceberg's Parquet writer (write.parquet.compression-codec).
• File size. Spark packs at most 128 MB into one read partition (spark.sql.files.maxPartitionBytes) and adds an estimated open cost for every file (spark.sql.files.openCostInBytes, 4 MB by default), so a lake of tiny files adds overhead to every scan. Compact small files, or let the table format's maintenance do it.
• Partitioning. Spark's built-in file sources discover key=value partition directories automatically and expose them as columns, so filters on those columns let Spark prune whole directories. Partition on columns that queries filter by, such as a date.
• Schema merging. It's off by default (spark.sql.parquet.mergeSchema) because the docs call it "a relatively expensive operation." Turn it on per read with the mergeSchema option when files have evolved.

(df.write
   .mode("overwrite")
   .partitionBy("event_date")
   .option("compression", "zstd")
   .parquet("/lake/events"))

recent = spark.read.parquet("/lake/events").where("event_date >= '2026-10-01'")

The compression option on the writer takes precedence over spark.sql.parquet.compression.codec, so one job can change its codec without touching the session default.

How Databricks approaches file formats

On Databricks the choice is mostly made for you. "Delta Lake is the default format for all operations on Databricks," and Delta Lake "extends Parquet data files with a file-based transaction log." Iceberg tables work too, on Parquet data files and Iceberg spec versions 1 through 3, whether Unity Catalog or a foreign catalog manages them. With Iceberg reads (UniForm), a Delta table also generates Iceberg metadata, so Iceberg clients read the same Parquet files and no second copy is needed. For Unity Catalog managed tables, Predictive Optimization schedules OPTIMIZE, VACUUM and ANALYZE automatically, taking over the upkeep a hand-built Parquet lake leaves to you.

FAQs

When does ORC make sense?

When your existing lake and tools already standardize on it. In Spark, ORC and Parquet are both columnar formats with filter pushdown and vectorized reads; Parquet is Spark's default data source, the file format under Delta Lake and Iceberg's default data file format.

Do I still need Parquet if I use Delta Lake or Iceberg?

Yes. Delta Lake's data files are Parquet, and Iceberg writes Parquet unless you set write.format.default to ORC or Avro.

How should I convert CSV landing data?

Read it once with an explicit schema, which skips the extra pass that inferSchema costs, then write it as Parquet or into a Delta Lake or Iceberg table.

Which compression codec should I choose?

Start with the defaults: snappy for Spark's Parquet writer and zstd for its ORC writer. If you change codecs, measure file size and read cost on your own data before changing a lake-wide default.