What's the best way to ingest large amounts of structured data from cloud storage into Databricks?

Summary

• The best way is Auto Loader inside Spark Declarative Pipelines (SDP), writing to streaming tables. Databricks recommends this combination for incremental ingestion from cloud object storage, and the pipeline manages the schema and checkpoint location for you.
• At large file counts, switch Auto Loader to file notification mode with file events. Databricks recommends it over the default directory listing mode for most workloads because it's more performant and scalable.
• Auto Loader records discovered files in its checkpoint, so each file is processed exactly once and a failed stream resumes where it stopped.
• COPY INTO still works for idempotent SQL loads, but Databricks recommends streaming tables for SQL users who need a more scalable file ingestion experience.
• Land and govern source files through Unity Catalog volumes or external locations, so access to raw files follows the same permission model as your tables.

What's the best way to ingest large amounts of structured data from cloud storage into Databricks?

Use Auto Loader in a Spark Declarative Pipelines (SDP) streaming table, with file notification mode and file events turned on. That combination avoids relying on directory listing for discovery, tracks the files it has processed in a checkpoint, and detects schema changes as columns appear. SQL-first teams can write the same pattern with CREATE STREAMING TABLE ... STREAM read_files, so they don't need Python.
Volume is what breaks naive file loading. Re-listing a bucket on every run, reprocessing files after a failure, or crashing on a new column all get worse as file counts grow. The choices below are about avoiding those failure modes.

What is Auto Loader?

Auto Loader is a Structured Streaming source called cloudFiles. Given an input directory, it processes new files as they arrive, with the option of also processing files already there. It reads JSON, CSV, XML, Parquet, Avro, ORC, text and binary files, including pre-compressed files, from cloud object storage and Unity Catalog volumes.

Key choices for large structured loads

Decision	Recommended option	Why
Ingestion engine	Auto Loader in SDP	Pipelines manage the schema and checkpoint location, plus orchestration and monitoring.
File discovery	File notification mode with file events	Databricks recommends it over directory listing for most workloads.
Landing zone	Unity Catalog volume or external location	Raw files get the same governance as tables.
Schema handling	Inference plus cloudFiles.schemaEvolutionMode	New columns are added to the schema, and data that doesn't match goes to the rescued data column.
Run mode	Triggered for batch, continuous for streaming	Pipelines run on a schedule or continuously.
SQL-only batch loads	COPY INTO	Retriable and idempotent, so previously loaded files are skipped.

How Databricks approaches large-scale file ingestion

Start with a streaming table. The incremental ingestion guide shows the pattern in SQL:

CREATE OR REFRESH STREAMING TABLE
  <table-name>
AS SELECT
  *
FROM
  STREAM read_files(
    '<path-to-source-data>',
    format => '<file-format>'
  )

The Python equivalent uses @dp.table with spark.readStream.format('cloudFiles') and .option('cloudFiles.format', '<file-format>'). Auto Loader processes new files as they arrive.
Turn on file events. In directory listing mode, Auto Loader finds new files by listing the input directory. That's the default and needs no permissions beyond access to your data, but as files accumulate, discovery slows down. File notification mode subscribes to storage events instead. To use the managed version, enable file events on the Unity Catalog external location and set cloudFiles.useManagedFileEvents to true. File events need Databricks Runtime 14.3 LTS and above, and you can switch detection modes across stream restarts without losing exactly-once guarantees.
Let the schema evolve. For typed formats such as Parquet and Avro, Auto Loader samples a subset of files and merges their schemas. For JSON, CSV and XML it infers every column as a string unless you enable type inference or add schema hints. When no schema is provided, the default addNewColumns mode adds new columns and stops the stream with an UnknownFieldException; the restart picks up the new schema, which is why Databricks recommends running these streams under Lakeflow Jobs so they restart automatically. The rescue mode never evolves the schema and never fails, sending new columns to the rescued data column.
Rely on the checkpoint. Auto Loader stores discovered file metadata in a RocksDB key-value store in the checkpoint location. That's what gives it exactly-once processing when writing to Delta Lake and lets it resume after failures without you managing state. Separately, the docs note that Auto Loader's discovery cost scales with the number of files ingested, not the number of directories they land in. If you must stay on directory listing, cloudFiles.cleanSource archives (MOVE) or deletes (DELETE) files after processing, so fewer files need listing.
Use COPY INTO where it fits. COPY INTO loads a file location into a Delta table, skips files it has already loaded, and supports schema inference and merging. For a more scalable file ingestion experience, Databricks recommends streaming tables for SQL users.

FAQs

Should I use Auto Loader or COPY INTO for a one-time bulk load?

Either works for a one-time load. Auto Loader is the better default when files will keep arriving, because the same pipeline handles the backfill and every later file. Databricks also documents Auto Loader for migrating or backfilling a table.

Do I need to define a schema up front?

No. Auto Loader infers the schema and stores it, and SDP manages the schema location for you. You can add schema hints to force specific column types.

Does Auto Loader process files in order?

No. Auto Loader doesn't guarantee the order in which files are discovered or processed in either detection mode, so design downstream logic to handle out-of-order arrivals.

Where should raw files live before ingestion?

Databricks recommends Unity Catalog volumes for governing non-tabular data, including landing areas for raw files. External volumes add governance to existing cloud storage paths without copying data.