What is Apache Parquet?

Summary

• Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval, with compression and encoding schemes for handling complex data in bulk.
• A Parquet file holds row groups, each with one column chunk per column, followed by file metadata, so a reader can locate and read only the columns it needs.
• Parquet is a file format, not a table format. Delta Lake extends Parquet data files with a transaction log, and Apache Iceberg tables can use Parquet, Avro, or ORC data files.
• Many languages and engines implement Parquet, but the project warns that not all implementations support the same features.
• On Databricks, Parquet is the underlying storage format for Delta Lake, the default table format, and you can also read and write Parquet files directly.

What is Apache Parquet?

Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides compression and encoding schemes to handle complex data in bulk, and it is supported in many programming languages and analytics tools. Databricks documentation notes that Parquet "allows query engines to read only the columns needed and skip irrelevant row groups."

How does Parquet organize data in a file?

Concept	What it is
Row group	A logical horizontal partitioning of the data into rows, with one column chunk for each column
Column chunk	The data for one column within a row group, contiguous in the file
Page	A division of a column chunk, conceptually an indivisible unit for compression and encoding
File metadata	Written after the data; readers read it first to find the column chunks they need

Parquet is built with nested data in mind and uses the record shredding and assembly algorithm described in the Dremel paper. LIST and MAP annotations encode nested types. Compression schemes can be specified per column, and the format supports several codecs, including SNAPPY, GZIP, BROTLI, ZSTD, and LZ4_RAW, which cover different points on the compression ratio and processing cost spectrum. Encodings include dictionary, run length, and delta encoding.

How does Parquet relate to Delta Lake and Apache Iceberg?

Format	What it is	Role of Parquet
Apache Parquet	A column-oriented data file format	Stores the data
Delta Lake	Open source software that extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling	The underlying storage format
Apache Iceberg	An open table format for analytic datasets	One of the immutable file formats its specification supports, with Avro and ORC

Databricks recommends Delta Lake when you need ACID transactions, schema enforcement, or time travel alongside Parquet's columnar performance. Delta Lake is the default format for tables on Databricks.

Which tools read and write Parquet?

The Parquet project lists open source implementations including parquet-java, Parquet C++, Parquet Go, Parquet Rust, Apache Impala, DuckDB, and fastparquet, a Python implementation. Its implementation status page also covers Polars and hyparquet for JavaScript. Because implementations differ in feature support, the project advises testing compatibility when you combine them.

How do I use Parquet with Spark and Databricks?

Spark reads and writes Parquet and preserves the schema, because Parquet files are self-describing:

peopleDF = spark.read.json("examples/src/main/resources/people.json")
peopleDF.write.parquet("people.parquet")
parquetFile = spark.read.parquet("people.parquet")

• Partitioning: In a partitioned table, data are usually stored in different directories, with partitioning column values encoded in the path. Spark's built-in file sources, including Parquet, discover partitions automatically. On Databricks, use partitionBy("year", "month") when writing.
• Schema merging: Set the mergeSchema read option to true to merge compatible schemas across Parquet files. It is off by default because merging is relatively expensive.
• Filtering: Parquet filter push-down is on by default in Spark (spark.sql.parquet.filterPushdown).
• Compression: Spark writes Parquet with snappy by default, and spark.sql.parquet.compression.codec accepts none, uncompressed, snappy, gzip, lzo, brotli, lz4, lz4_raw, and zstd.
• SQL access: Use read_files with format => 'parquet' to query Parquet files directly from cloud storage without creating a table. Specify a schema when reading to avoid the overhead of schema inference.
• Moving to Delta Lake: Databricks offers a one-time conversion of Parquet tables to Delta Lake and incremental cloning of Parquet tables.

FAQs

Is Parquet only for Databricks?

No. Parquet is an Apache Software Foundation project with implementations in Java, C++, Go, Rust, and other languages. On Databricks, it is the underlying storage format for Delta Lake.

Does Parquet support nested data?

Yes. Parquet encodes nested columns with the Dremel encoding, using definition and repetition levels, and annotates lists and maps with LIST and MAP.

Can I add columns to existing Parquet data?

Spark can merge compatible schemas across Parquet files when mergeSchema is on. For Delta Lake tables on Databricks, schema evolution updates a table schema without rewriting data.

How much will Parquet compress my data?

There is no fixed ratio to quote. The format offers several codecs that cover different points on the compression ratio and processing cost spectrum, and you choose the codec when you write.

Can I convert CSV to Parquet?

Yes. In Spark, load the CSV file with the csv format and save the DataFrame with the parquet format. Spark documents that DataFrames loaded from any data source type can be converted into other types.