What is the best way to ingest data from SaaS sources like Salesforce into Databricks?

Summary

• The best way is a managed SaaS connector in Lakeflow Connect. The Salesforce connector has been generally available since April 2, 2025, and it handles authentication, incremental reads, schema evolution and retries for you.
• Each pipeline lands objects as streaming tables in Unity Catalog, runs on serverless compute, and can be built in the UI, through the API, or as code with Declarative Automation Bundles.
• Incremental ingestion relies on a cursor column. The connector picks SystemModstamp, LastModifiedDate, CreatedDate or LoginTime, in that order of preference, and fetches only changed records.
• Soft deletes flow through on the next sync, SCD type 2 history is supported, and you can select columns and filter rows at ingestion. Hard deletes need a full refresh.
• For SaaS applications without a managed connector, use a community connector or build a custom one. Both are in Beta.

What is the best way to ingest data from SaaS sources like Salesforce into Databricks?

Use the managed Salesforce connector in Lakeflow Connect. You create a Unity Catalog connection, pick the objects you want, choose a destination schema and set a schedule. The connector handles API authentication, incremental reads, schema changes and retries, and writes the data to governed streaming tables, so there's no API client code for you to maintain.
SaaS APIs change, enforce usage limits and paginate. A managed connector moves that maintenance off your team.

What a SaaS ingestion approach should handle

Capability	Why it matters
Authentication	OAuth flows and token refresh should be managed, with credentials stored centrally.
Incremental reads	Pulling full objects on every run wastes API capacity and compute.
API selection	Large initial loads and small incremental loads suit different API types.
Schema evolution	Source admins add custom fields. New columns should appear without a rebuild.
Deletes and history	You need deleted records reflected and, for some tables, full change history.
Governance	Ingested CRM data can hold personal data, so access control should apply from the first write.
Deployment as code	Pipelines should move through dev, staging and production with CI/CD.

How Databricks approaches SaaS ingestion

Managed connectors for enterprise applications. Lakeflow Connect's SaaS connectors each handle source-specific authentication, incremental reads, schema evolution and automated retries. A connector has three parts: a connection (a Unity Catalog securable object holding the credentials), an ingestion pipeline on serverless compute, and destination streaming tables. The Salesforce connector became generally available on April 2, 2025, per the release notes.
How the connector uses the source APIs. Per the connector FAQ, the connector uses both the source's bulk API and its REST API, choosing per pipeline update based on how much data it must ingest. Large initial loads usually go through the bulk API; small incremental loads go through REST. The goal is to limit load on the source APIs. Failed requests retry with exponential backoff.
Incremental by default. The connector picks a cursor column (SystemModstamp, then LastModifiedDate, CreatedDate and LoginTime) and reads only records that changed. Objects with none of those columns can't be ingested incrementally. Formula fields are fully snapshotted on each run by default; incremental formula field ingestion is available in Beta.
History, filtering and schema change. The connector supports SCD type 2, API-based column selection and row filtering, and automatic handling of new and deleted columns. A renamed column is treated as a new column plus a deleted one. Data type changes aren't evolved automatically.
Pipelines as code. The docs show a Declarative Automation Bundles definition like this:

resources:
  pipelines:
    pipeline_sfdc:
      name: salesforce_pipeline
      catalog: my_catalog
      schema: my_schema
      ingestion_definition:
        connection_name: <salesforce-connection>
        objects:
          - table:
              source_schema: objects
              source_table: Account
              destination_catalog: my_catalog
              destination_schema: my_schema

A matching job definition sets the refresh frequency with a pipeline_task. When you add a schedule in the UI, Lakeflow Connect creates a Lakeflow Jobs job for it.
When there's no managed connector. Community connectors cover more sources; they're built and maintained by the community and not backed by Databricks SLAs. You can also build a custom connector and run it in your own workspace. Both features are in Beta.

Use cases

• Customer 360: combine CRM data with other sources to measure campaign performance and score leads.
• Churn prediction: combine CRM customer order data with customer interactions across other channels to predict churn.
• Historical analysis: keep SCD type 2 history of changing records to analyze them over time.

FAQs

What authentication does the Salesforce connector use?

OAuth U2M and OAuth with mutual TLS (mTLS). mTLS became generally available on August 20, 2026, and suits API-only users or organizations that require it. When you create or reauthorize a connection, a source admin may need to install the Databricks connected app or grant you heightened permissions.

How are deletes handled?

Soft deletes are applied at the next sync. With history tracking on, the original row is marked inactive through the __END_AT column. Hard deletes, and records purged from the recycle bin before the next run, require a full refresh.

How many objects can one pipeline ingest?

Databricks recommends limiting one pipeline to 250 tables. Create additional pipelines for more objects.

Does the connector support real-time ingestion?

No. It runs on a schedule you set, so choose a frequency that balances freshness against cost.