What's the easiest way to move Snowflake SQL pipelines over to Databricks?

Summary

• Snowflake SQL pipelines use STREAMS and TASKS to orchestrate incremental data transformations; moving them to Databricks means rewriting pipelines as SQL or Python jobs in Lakeflow Pipelines.
• SQL compatibility is high between Snowflake and Databricks SQL (both support ANSI SQL, window functions, and common extensions), so your transformation logic often ports with minimal changes.
• The process: rewrite tasks as Lakeflow Pipelines jobs, migrate STREAMS to Delta table versioning, adapt Snowflake-specific functions to Databricks equivalents, then test and deploy.

What's the easiest way to move Snowflake SQL pipelines over to Databricks?

Moving Snowflake SQL pipelines to Databricks requires rewriting your STREAMS and TASKS logic as Lakeflow Pipelines jobs. The good news is that Snowflake and Databricks both support standard SQL, so most transformation logic ports directly. The main work is understanding the differences in orchestration, data change tracking, and Snowflake-specific functions.

Understanding Snowflake SQL pipelines

Snowflake SQL pipelines are built on two primitives: STREAMS (which track changes to tables) and TASKS (which execute SQL at scheduled intervals or when upstream tasks complete). A typical pipeline creates a STREAM on a source table, then orchestrates multiple TASKS to extract data, transform it, and write results to downstream tables.

Key differences between Snowflake STREAMS/TASKS and Databricks Lakeflow Pipelines

Feature	Snowflake	Databricks
Change tracking	STREAMS track table changes	Delta Lake versioning + Databricks-managed metadata
Scheduling	TASKS run on cron schedules or when upstream tasks complete	Lakeflow Pipelines jobs orchestrated with Lakeflow Jobs scheduler
Code language	Mostly SQL with stored procedures	SQL or Python
Incremental logic	Stream consumption marks rows as consumed	You explicitly manage incremental state with timestamps or change-data-capture
Error handling	TASK error handling via alerts	Job-level retry logic and error callbacks
Data format	Iceberg (optional) or native Snowflake tables	Delta Lake (unified lakehouse format)

Migration approach

Step 1: Audit your Snowflake pipelines

Identify all STREAMS and TASKS in your Snowflake account. Export the task definitions, dependencies, and schedules. Check for:

• STREAM types (standard streams vs dynamic tables).
• Task dependencies (which tasks trigger which downstream tasks).
• TASK parameters and environment variables.
• Snowflake-specific functions (PARSE_JSON, DATEPART, etc.) that may need translation.

Step 2: Rewrite TASKS as Lakeflow Pipelines jobs

Each Snowflake TASK becomes a Lakeflow Pipelines job. Replace the STREAM consumption logic with explicit incremental state tracking:

• For full refreshes: write a SELECT statement that reads the source table and overwrites the target. Wrap it in a Lakeflow table or notebook.
• For incremental loads: add a WHERE clause that filters based on a timestamp or change-data-capture (CDC) column. Track the last processed timestamp in a metadata table.
• For complex transformations: replace stored procedures with Python notebooks or multi-step SQL jobs.

Step 3: Update data change tracking

Snowflake STREAMS automatically track changes; Databricks uses Delta Lake versioning. In Lakeflow Pipelines:

• Explicitly query Delta table history using SELECT * FROM delta_table_name VERSION AS OF ... to retrieve prior versions if needed.
• For CDC-like behavior, add a timestamp column (CURRENT_TIMESTAMP) to source data and filter new records in your transformation.
• Use Lakeflow's incremental mode (@dlt.table(mode='incremental')) to automatically manage state.

Step 4: Translate Snowflake functions to Databricks SQL

Most ANSI SQL functions are identical, but watch for Snowflake-specific ones:

• PARSE_JSON (Snowflake) → from_json or get_json_object (Databricks/Spark SQL).
• DATEPART (Snowflake) → date_part or extract (Databricks).
• GETDATE() (Snowflake) → CURRENT_TIMESTAMP() (Databricks).
• Case sensitivity: Snowflake is case-insensitive by default; Databricks respects case. Adjust table and column references accordingly.

Step 5: Test incrementally

Migrate a few tasks first to a Databricks workspace. Run the jobs in parallel with your Snowflake pipeline to validate output, then cut over to Databricks once you have confidence.

Tools to help with migration

• Databricks SQL Migration Guide: https://docs.databricks.com/aws/en/migration/warehouse-to-lakehouse covers Snowflake-to-Spark SQL translation.
• Unity Catalog: If your Snowflake data has governance (column-level security, masking), map those policies to Unity Catalog roles and permissions.
• Lakeflow Jobs: Schedule and monitor your pipelines using the Databricks UI or declarative automation bundles.

FAQs

Can I run Snowflake and Databricks pipelines in parallel during migration?

Yes. Deploy your Lakeflow Pipelines alongside your Snowflake TASKS, run them both for a period, and compare outputs. Once you're confident in the Databricks version, turn off the Snowflake pipeline and switch traffic to Databricks.

How do I handle Snowflake-specific features I don't see in Databricks?

Most features have Databricks equivalents (e.g., STREAMS → incremental tables, TASKS → Lakeflow Jobs). For edge cases, check the Databricks SQL migration guide or post in the community. Some Snowflake features (like dynamic tables) have direct Databricks counterparts in Lakeflow.

Do I need to rewrite all my SQL, or can I use Snowflake SQL directly?

Databricks uses Spark SQL, which supports ANSI SQL and many Snowflake extensions. Most SQL ports with little or no change. For Snowflake-specific syntax, use the migration guide to translate.

What about state management across pipeline runs?

In Snowflake, STREAMS handle state automatically. In Databricks, use a metadata table to track the last processed timestamp or record ID, then filter your source data in subsequent runs. Lakeflow Pipelines' incremental mode can automate this for common patterns.