How do you ingest data into Databricks?

Summary

• Databricks unifies file-based, database CDC, streaming, and SaaS data ingestion through LakeFlow, eliminating the need for separate toolchains.
• Auto Loader, COPY INTO, and Structured Streaming each address different ingestion patterns while writing to open formats governed by Unity Catalog.
• Best practices include separating raw and curated layers, using incremental loading by default, and centralizing governance at ingestion time to ensure data quality.

How to ingest data into Databricks

Getting data from source systems into a central analytics platform is the first step in any data pipeline, and often the most fragile. Teams juggle separate tools for batch loads, streaming feeds, database replication, and file ingestion, each with its own scheduling and error handling.
The result is brittle handoffs, stale data, and metrics that teams cannot agree on. A reliable ingestion strategy must handle diverse sources, changing schemas, and both real-time and batch workloads from a single foundation.

Why unified data ingestion matters

Fragmented ingestion stacks create three recurring problems:

• Duplicate pipelines: Separate batch and streaming code doubles maintenance effort.
• Stale or inconsistent data: Handoffs between tools introduce latency and trust gaps.
• Governance gaps: Data lands in different stores with different permissions, making lineage and auditing difficult.

The financial stakes are significant. According to Gartner, poor data quality costs organizations an average of $12.9 million per year, much of it traceable to fragmented ingestion that introduces inconsistencies before data ever reaches an analyst.
Solving these problems requires consolidating pipelines so every workload writes to one open, governed foundation. A data quality management strategy that starts at ingestion time can prevent costly downstream issues.

Common data ingestion methods

Before choosing a tool, understand the four broad ingestion patterns most teams encounter.

• File-based ingestion: Loading CSV, JSON, Parquet, or other files from cloud object stores like S3, ADLS, or GCS.
• Database replication: Capturing changes from relational databases such as PostgreSQL or MySQL using change data capture (CDC).
• Streaming ingestion: Reading continuous event data from message buses like Apache Kafka, Amazon Kinesis, or Azure Event Hubs.
• SaaS application ingestion: Pulling data from tools like Salesforce or Workday via managed connectors.

Each pattern has different latency, volume, and schema-change characteristics. The best ingestion strategies handle all four without requiring separate toolchains.

How Databricks simplifies data ingestion with LakeFlow

Databricks unifies real-time and batch ETL directly in the data lakehouse. Lakeflow is the unified orchestration layer for both batch and streaming pipelines.

• Lakeflow Connect: Ingest from relational databases and SaaS applications using managed connectors with CDC.
• Auto Loader: Process new data files incrementally as they arrive in cloud storage.
• COPY INTO: Load data from a file location into a Delta table using a retriable, idempotent SQL command.

Unity Catalog provides one catalog for all data, managing Delta Lake, Apache Iceberg™, and Parquet with a single set of permissions, lineage, and business definitions.

Choosing the right ingestion method

Method	Best for	Key trait
Lakeflow Connect	Databases, SaaS apps	Managed connectors with CDC
Auto Loader	Cloud storage files at scale	Incremental streaming file processing
COPY INTO	Bulk or ad-hoc SQL loads	Idempotent SQL command
Structured Streaming	Kafka, Kinesis, Event Hubs	Continuous low-latency streams

All four methods write to open formats, Delta Lake, Apache Iceberg, and Parquet, as first-class citizens, not bolt-ons. Lakehouse storage ensures these open formats are managed efficiently under one platform.

Best practices for data ingestion

1. Separate raw and curated layers. Land data in a raw zone first, then validate and transform downstream.
2. Use incremental loading by default. CDC or file-based checkpointing reduces cost and latency versus full reloads.
3. Enable schema evolution selectively. Allow new columns in raw layers; enforce strict schemas downstream.
4. Centralize governance early. Apply access controls and lineage tracking at ingestion time.
5. Monitor pipeline health. Track file counts, row counts, and processing latency. Alert on schema drift early. Tools like Lakehouse Monitoring can help automate observability across your pipelines.

FAQs

What are the different methods for ingesting data into a lakehouse?

The primary methods are file-based ingestion, database replication via CDC, streaming from message buses, and SaaS connectors. Databricks unifies all four through Lakeflow so every pipeline writes to one governed, open data layer.

How do you use auto loader to incrementally ingest files?

Auto Loader provides a Structured Streaming source called cloudFiles. Given an input directory on cloud storage, it automatically processes new files as they arrive.

How do you set up a LakeFlow pipeline for data ingestion?

Write declarative Python or SQL that defines your source and target. Lakeflow pipelines handle both batch and streaming workloads in one orchestration layer.

How do you ingest streaming data using structured streaming?

Structured Streaming reads continuous data from Kafka, Kinesis, or Event Hubs and writes to Delta tables. Lakeflow orchestrates these streaming pipelines alongside batch workloads.

How do you connect to cloud storage like s3, adls, or gcs for data ingestion?

Configure access credentials for your cloud provider, then use Auto Loader or COPY INTO to read files. Unity Catalog manages credentials and permissions across all storage locations.

How do you use the copy into command to load data from external sources?

COPY INTO is an idempotent SQL command that loads files into a Delta table and skips previously loaded files. For millions of files, Auto Loader is more efficient.

What file formats are supported for data ingestion, and which is most efficient?

Common source formats include CSV, JSON, Avro, ORC, Parquet, and text files. Parquet and Delta are the most efficient because they are columnar, compressed, and support predicate pushdown.

How do you ingest data from relational databases like PostgreSQL or mysql?

Use CDC-based replication to capture inserts, updates, and deletes. Lakeflow Connect provides managed connectors for MySQL, PostgreSQL, Oracle, and SQL Server.

What are best practices for handling schema evolution during data ingestion?

Enable schema evolution only in raw ingestion layers where new columns are expected. Validate schemas downstream to enforce data contracts. Auto Loader can infer and evolve schemas automatically.

How do you schedule and automate data ingestion workflows?

Lakeflow provides built-in orchestration. You can run ingestion on custom schedules or event-driven triggers, and add downstream tasks to the same workflow.

Start ingesting data on a unified lakehouse foundation

Lakeflow brings batch, streaming, database, and SaaS ingestion together on one open, governed platform. With Unity Catalog providing lineage, permissions, and business definitions across every pipeline, data stays fresh, consistent, and ready for analytics and AI, all from a single trusted source. Explore the data lakehouse to see how Databricks unifies ingestion, governance, and analytics on one platform.