How do I migrate to Databricks?

Summary

• A Databricks migration moves data, SQL and Spark code, pipelines, schedules, and access policies onto one platform built on Apache Spark, Delta Lake, and Unity Catalog.
• Plan it in phases: assess the estate, convert code, load data, rebuild pipelines, recreate governance, then reconcile results against the source before cutover.
• The agentic code converter (Beta) uses Genie Code to convert legacy SQL into Databricks SQL or DataFrame notebooks, validating the output and retrying sections that fail.
• Lakebridge, a Databricks Labs toolkit, adds a Profiler and Analyzer to size the work, SQL transpilers, and Reconcile to compare migrated data with the source.

How do I migrate to Databricks?

Migrate in phases and let tooling handle repetitive conversion. Inventory what you run, convert SQL with the agentic code converter or Lakebridge, land data as Delta tables governed by Unity Catalog, and rebuild ingestion and scheduling with Lakeflow Pipelines and Lakeflow Jobs. Then reconcile the migrated data against the source before you switch consumers over. Databricks' guidance is that most workloads, queries, and dashboards defined in enterprise data warehouses can run with minimal code refactoring once admins have completed the initial data migration and governance configuration.

What does a Databricks migration involve?

You move the data, the code that transforms it, the schedules that run it, the access rules that protect it, and the dashboards that read it. Databricks describes the goal as unifying your data ecosystem, so analysts, data scientists, and engineers work against the same tables. Plan for a few documented differences from a traditional warehouse:

• Unity Catalog uses a three-level catalog.schema.table namespace.
• Primary key and foreign key constraints are informational only.
• Native data types might differ slightly from the source, so settle numeric precision first.
• Spark SQL is mostly ANSI-compliant, but some differences in behavior might exist.

Which migration phases should you plan for?

Phase	What happens	Databricks tooling
Assess	Size the SQL code and its orchestration, and flag likely issues	Lakebridge Profiler and Analyzer
Convert	Translate legacy SQL and adapt Spark jobs	Agentic code converter (Beta), Lakebridge transpilers
Load	Land data as Delta tables, or query it in place during the transition	Lakeflow Connect, Auto Loader, Lakehouse Federation
Rebuild pipelines	Recreate transformations, schedules, and dependencies	Lakeflow Pipelines, Lakeflow Jobs, dbt, Airflow
Govern	Recreate permissions and row-level rules	Unity Catalog, row filters, column masks, SCIM
Validate and cut over	Compare outputs with the source, then repoint consumers	Lakebridge Reconcile, query profile

How do you convert legacy SQL and Spark code?

The agentic code converter is in Beta; a workspace admin turns it on from the Previews page. It uses Genie Code to convert SQL from several common warehouse dialects. You organize files in a migration project and pick a target: ANSI-compliant Databricks SQL, or notebooks that use the DataFrame API. It validates each result and iteratively fixes errors before writing to a workspace folder. Upload Lakebridge Analyzer output to the project to see code complexity and a dependency graph of files and tables that must move together.
Lakebridge is a Databricks Labs project, provided as-is without formal support SLAs. Its transpilers are BladeBridge (a wide range of dialects plus some ETL and orchestration code), Morpheus (a smaller set of dialects, with experimental dbt support), and Switch (LLM-powered conversion to notebooks).
Most Spark jobs need only a few changes. Databricks recommends writing Delta Lake instead of Parquet or ORC, using SparkSession.builder().getOrCreate() instead of creating sessions, removing sys.exit() and sc.stop(), and dropping custom executor or JVM settings.

How do you move ETL pipelines and orchestration?

• dbt: the dbt-databricks adapter connects dbt Core to a SQL warehouse or cluster, and dbt Core projects can run as a Lakeflow Jobs task.
• Airflow: the open source apache-airflow-providers-databricks package (not directly supported by Databricks) includes DatabricksRunNowOperator, which Databricks recommends for triggering existing jobs, and DatabricksSubmitRunOperator.
• File-driven logic: use Structured Streaming with Lakeflow Jobs, and Auto Loader for cloud object storage.
• Native rebuild: Lakeflow Pipelines builds batch and streaming pipelines in SQL or Python, with expectations that drop invalid records or fail the update. Lakeflow Jobs runs work on a schedule, on file arrival, on table updates, or continuously.

How do you recreate governance and access?

Unity Catalog is the unified governance layer for data and AI built into Databricks: it enforces access control, tracks lineage, and logs activity for auditing. Rebuild row-level security with row filters and column masks, both written as SQL user-defined functions. Sync users and groups from your identity provider with SCIM or automatic identity management.

How do you validate results and cut over?

Lakebridge Reconcile compares the source system with the Databricks target and identifies discrepancies at the row, column, and schema level. During the parallel run, Lakehouse Federation gives governed, read-only access to external data through Unity Catalog foreign catalogs, so you can query the source from Databricks without copying it. When a migrated query is slow, its query profile shows bottlenecks such as exploding joins or full table scans.

What are common questions about migrating to Databricks?

How long does a migration take?

It depends on code volume, complexity, and orchestration. That's why assessment comes first: the Lakebridge Analyzer reports code size and complexity and flags potential issues.

Do I need to rewrite all my queries?

No. SQL workloads usually need very little refactoring, depending on how much the code relies on system-specific protocols. For SQL projects, the converter shows a side-by-side diff of each proposed change.

Is the agentic code converter generally available?

No. It's in Beta, a workspace admin must enable it, and per-batch and per-script limits apply.