How do I pull data into Databricks from operational databases like SQL Server?

Summary

• Use the managed SQL Server connector in Lakeflow Connect, which has been generally available since August 26, 2025. It tracks changes in the source, applies them incrementally and writes to streaming tables governed by Unity Catalog.
• The connector uses change tracking or change data capture (CDC) in the source. Databricks recommends change tracking for tables with a primary key, because it puts less load on the database, and CDC for tables without one.
• The standard setup has an ingestion gateway that runs continuously on classic compute, a staging volume, and an ingestion pipeline on serverless compute that you schedule.
• When CDC isn't an option, query-based connectors (generally available) read changed rows using a cursor column, with no gateway or staging volume.
• If you only need to query the data, not copy it, Lakehouse Federation offers a zero-copy connector for SQL Server.

How do I pull data into Databricks from operational databases like SQL Server?

Configure the source for change tracking or CDC, create a Unity Catalog connection, then build a gateway and ingestion pipeline with the managed SQL Server connector in Lakeflow Connect. The first run takes a snapshot. Later runs apply only the changes, so the Delta tables stay current without full reloads or hand-written extract jobs.
Pulling from an operational database is a balancing act. You want fresh data in the lakehouse, but you can't let ingestion slow down the application that depends on that database.

What is change data capture?

Change data capture (CDC) is a way of reading a database's record of changes instead of re-querying whole tables. SQL Server offers two mechanisms. Change tracking records that a row changed, needs a primary key, and is lightweight on the source. CDC records every operation with history over time, doesn't need a primary key, and can have more of an impact on source performance. If both are enabled, the connector uses change tracking.

Your options at a glance

Approach	How it works	When to use it
Managed CDC connector	Gateway captures snapshots and changes into a staging volume; a serverless pipeline applies them to streaming tables.	Ongoing replication that tracks source changes and, with CDC, supports SCD type 2 history.
Query-based connector	Scheduled queries pick up rows whose cursor column passed the last high-water mark.	CDC can't be enabled, or periodic latest-state copies are enough.
Lakehouse Federation	Queries the database in place with a zero-copy connector.	You only need to query the data, and copying it isn't worth it.
Custom code	SQL and Spark APIs in Spark Declarative Pipelines (SDP) or Structured Streaming.	Requirements the managed options don't cover.

How Databricks approaches database ingestion

Prepare the source. Create a database user dedicated to Databricks ingestion, enable change tracking or CDC, and configure firewall settings if needed. Databricks provides a utility objects script for permissions and enablement.
Create the connection. The connection is a Unity Catalog securable object that stores credentials, which can only be retrieved by users with the right permissions. The connector supports OAuth (user-to-machine and machine-to-machine) and username and password. It connects over TLS using JDBC, and new pipelines validate the server's TLS certificate.
Run the gateway and pipeline. Per the SQL Server connector docs, the ingestion gateway runs continuously on classic compute so it captures changes before the source truncates its change logs. It writes to a staging volume, which also helps with failure recovery and is purged automatically after 30 days. The ingestion pipeline runs on serverless compute on whatever schedule you set. Because the gateway is always on, you pay for its compute even when the pipeline is idle, so size it for your initial snapshot.
As code, the gateway and pipeline are two resources in a Declarative Automation Bundle. A trimmed version of the docs example:

gateway_definition:
  connection_name: ${var.connection_name}
  gateway_storage_catalog: ${var.staging_catalog}
  gateway_storage_schema: ${var.staging_schema}

ingestion_definition:
  ingestion_gateway_id: ${resources.pipelines.gw_pipeline.id}
  table_configuration:
    scd_type: 'SCD_TYPE_1'
  objects:
    - schema:
        source_catalog: ${var.source_database}
        source_schema: ${var.source_schema}
        destination_catalog: ${var.dest_catalog}
        destination_schema: ${var.dest_schema}

Choose history tracking. SCD_TYPE_1 overwrites rows with the latest values. SCD_TYPE_2 keeps history with __START_AT and __END_AT columns, and the docs note it requires CDC on the source, because change tracking doesn't support it.
Plan for schema change. New and deleted columns evolve automatically. New tables are picked up if you ingest the entire schema. Column renames and data type changes aren't handled automatically; a rename needs a full refresh.
Connect the network. The gateway runs in your workspace VPC or VNet and must reach the database. Supported paths include VPN, private interconnects, VPC or VNet peering and public endpoints, so on-premises databases can work too when bandwidth is sufficient.
Fall back to query-based ingestion. Query-based connectors became generally available on May 29, 2026, and SQL Server is a supported source. They track a monotonically increasing timestamp or integer column, run on serverless compute by default, and don't need a gateway or staging volume. The trade-off: queries hit source tables directly, which can add load, and they capture only the latest state of changed rows between runs.

FAQs

Which source database versions are supported?

Change tracking requires version 2012 or above. CDC requires version 2012 service pack 1 (SP1) cumulative update 3 (CU3) or above, and versions earlier than 2016 also need Enterprise Edition.

How many tables can one pipeline ingest?

The docs list a maximum of 250 tables per pipeline. Split larger sources across multiple pipelines.

Can I run the ingestion pipeline continuously?

The gateway runs continuously, but the ingestion pipeline in the standard setup runs on a schedule. Schedule it with Lakeflow Jobs at the frequency your latency and cost needs require.

Does the same approach work for other operational databases?

Yes. Lakeflow Connect provides managed CDC connectors for several relational databases, with the same gateway, staging and pipeline structure.