How do I migrate off Databricks to open-source Spark, and back?

Summary

• Table data moves with the least effort: Delta Lake and Apache Iceberg are open table formats. Check which Delta table features your tables use first, because open-source Delta clients choose which features they support.
• Spark code carries over with small edits: open-source Spark needs the SparkSession and job shutdown handled in code. Photon needs no code changes.
• Pipelines built on Spark Declarative Pipelines (SDP) stay portable. Lakeflow-only extras and Lakeflow Jobs need replacements, for example Apache Airflow.
• Coming back, register your storage as Unity Catalog external tables.

How do I migrate off Databricks to open-source Spark, and back?

Treat the move as three tracks: data, code, and the services around them (jobs, dashboards, permissions). Delta Lake, Apache Iceberg, and the Spark APIs are open, so data and DataFrame code need the least rewriting. The work is replacing Databricks services and testing table features. Stage it: compute first with data left in place, then the catalog, then orchestration.

What carries over to open-source Spark?

• Table data. Delta Lake is open source software that extends Parquet data files with a file-based transaction log, and the log's protocol is open, so any system can read it. Apache Iceberg is an open standard for analytic tables that any processing engine can use.
• DataFrame and SQL code. Apache Spark powers Databricks compute clusters and SQL warehouses, so expect small edits, not a rewrite.
• Declarative pipelines. Lakeflow Pipelines are built on Spark Declarative Pipelines (SDP), and Databricks says the transformation code stays portable to other SDP runtimes. SDP arrived in Apache Spark 4.1.0. Install it with pip install pyspark[pipelines] and run it with spark-pipelines run.
• Photon. It needs no code changes. For supported operations it replaces the JVM-based Spark SQL execution engine with a native C++ runtime. On open Spark, queries run on Spark's own engine, so retest run times.

What do I have to change or replace?

Layer	On Databricks	On open-source Spark
Session and shutdown	A SparkSession is created for each notebook or job, and jobs end on their own	Create the session with SparkSession.builder and keep awaitTermination() on streaming queries
Delta access	Built in	Add the delta-spark package and the two session settings shown below
Pipeline extras	AUTO CDC, data quality expectations, a queryable event log, continuous mode	Not part of SDP, so rewrite them as Structured Streaming code you maintain
Orchestration	Lakeflow Jobs, workflow automation for Databricks	Rebuild the task graph in an open orchestrator such as Apache Airflow
Dashboards	AI/BI Dashboards	Rebuild in a BI tool that connects through Spark's Thrift JDBC/ODBC server
Permissions	Unity Catalog privileges	Recreate grants in open-source Unity Catalog or your storage layer
File paths	DBFS root and mounts, both deprecated	Use object-store URIs

Can open-source Spark read a Delta table written on Databricks?

Yes, when your client supports the table's protocol. Reader features such as column mapping, deletion vectors, liquid clustering, and type widening block reads in clients without them, while writer features such as CHECK constraints block only writes. Open-source Delta clients choose how to implement support for given features. Run DESCRIBE DETAIL to see each table's minReaderVersion and minWriterVersion, and check the Delta releases page for the Spark versions each Delta Lake release supports. For external tables, read by path:

import pyspark
from delta import *

builder = pyspark.sql.SparkSession.builder.appName("MyApp") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
spark = configure_spark_with_delta_pip(builder).getOrCreate()
df = spark.read.format("delta").load("s3a://<bucket>/<table-path>")

Unity Catalog managed tables are different. Path-based access is not supported (except in Compatibility Mode) because it bypasses Unity Catalog access controls and might cause data corruption or loss. Read them by name through Unity Catalog's open APIs: the Unity REST API for Delta clients, or the Iceberg REST catalog endpoint (/api/2.1/unity-catalog/iceberg-rest) for Iceberg clients, which also reads Delta tables with Iceberg reads enabled. For the Iceberg endpoint, first enable external data access on the metastore and grant EXTERNAL USE SCHEMA.

How do I move back to Databricks?

1. Register Delta data as external tables. Configure an external location for the storage, then run CREATE TABLE <catalog>.<schema>.<table-name> LOCATION 's3://<bucket-path>/<table-directory>'. You need the CREATE EXTERNAL TABLE privilege on that external location.
2. Check Iceberg tables against Databricks limits. Databricks supports Iceberg format versions 1, 2, and 3 with Parquet files only, and for v2 it does not support position or equality deletes. Tables in catalogs outside Unity Catalog are read-only foreign tables.
3. Adapt the code. SparkSession creation, sys.exit(), sc.stop(), and awaitTermination() calls can be removed, because Databricks handles them. Keep SparkSession.builder().getOrCreate() to keep testing locally. Recompile against your Databricks Runtime's libraries, and set only the Spark configurations you need, since custom executor or JVM settings can reduce performance.
4. Recreate orchestration. Use Lakeflow Jobs, or keep Airflow and trigger Databricks jobs with DatabricksRunNowOperator from the open source Databricks provider.
5. Match Spark versions. The Databricks Runtime release notes list each runtime's Apache Spark version, for example 4.1.0 for 18 LTS. Using the same version avoids upgrade-related syntax changes.

FAQs

Do I have to convert Delta tables to Parquet to leave Databricks?

No. The Delta transaction log protocol is open and Delta Lake runs on S3, ADLS, GCS, and HDFS, so converting is optional.

What happens to my Unity Catalog permissions?

Plan to recreate them. An open-source Unity Catalog under the Apache 2.0 license keeps users and privileges in its own user database. Its README says the APIs are evolving and should not be assumed stable, and its Spark integration needs at least Apache Spark 3.5.3 and Delta Lake 3.2.1.

Can I keep my Spark Connect client code?

Mostly. Databricks Connect extends Spark Connect with additions for Databricks compute modes and Unity Catalog, so change the client library and connection settings. Spark Connect does not support RDD APIs and has no built-in authentication, so use an authenticating proxy.