How do I migrate from Hadoop or Hive to Spark?

Summary

• Migrate in layers. Run Spark where your data already lives (HDFS, YARN, the Hive metastore), move MapReduce and HiveQL logic next, then replace the scheduler and optionally convert tables to Delta Lake or Apache Iceberg.
• Spark can read existing Hive tables in place. With Hive support enabled it connects to your Hive metastore, and the Spark docs say that deploying against an existing Hive warehouse needs no change to the metastore, data placement or partitioning.
• Spark SQL supports the vast majority of Hive features, but the Spark migration guide lists differences: views need explicit column aliases, a few functions return different results, and some Hive UDF APIs are unsupported.
• Apache Oozie retired in February 2025. Apache Airflow's Spark provider can schedule spark-submit jobs on YARN, and on Databricks, Lakeflow Jobs orchestrate workloads.

How do I migrate from Hadoop or Hive to Spark?
Migrate in layers. First run Spark on the cluster and storage you already have, then move MapReduce and HiveQL logic, then replace the scheduler and decide whether to convert tables to an open table format. Spark reads Hive tables in place, so you can start without copying data. Check the compatibility differences below before you cut over.

How does Spark run on an existing Hadoop cluster?

Hadoop MapReduce splits input into independent chunks for map tasks, sorts the map outputs and feeds them to reduce tasks. YARN separates resource management from job scheduling and monitoring. Spark is a unified analytics engine for large-scale data processing, built on an optimized engine that supports general execution graphs. It uses Hadoop's client libraries for HDFS and YARN.

1. Run Spark on YARN. Set HADOOP_CONF_DIR and submit with spark-submit --master yarn. Spark finds the cluster from HADOOP_CONF_DIR or YARN_CONF_DIR.
2. Rewrite MapReduce logic as Spark SQL, DataFrame operations or RDD operations. reduceByKey merges the values for each key and, like a MapReduce combiner, merges locally on each mapper before sending results to a reducer.
3. Compare outputs from the old and new jobs before you retire the MapReduce jobs.

How do I read Hive tables from Spark?

Hive's hive.execution.engine accepts mr, tez or spark, and the Hive docs mark mr as deprecated since Hive 2. The metastore holds the table definitions, so Spark connects to it and reads the data files where they are:

from pyspark.sql import SparkSession

spark = (SparkSession.builder
    .appName("hive-migration")
    .enableHiveSupport()
    .getOrCreate())

spark.sql("SELECT * FROM my_hive_table").show()

Put hive-site.xml in Spark's conf/ directory. Without it, Spark creates a local metastore_db in the current directory. By default spark.sql.hive.metastore.version is 2.3.10, and the available options are 2.0.0 through 2.3.10, 3.0.0 through 3.1.3 and 4.0.0 through 4.1.0. For JDBC and ODBC clients, Spark's Thrift JDBC/ODBC server (started with ./sbin/start-thriftserver.sh) corresponds to HiveServer2 in built-in Hive, and the Spark docs say it is designed to be "out of the box" compatible with existing Hive installations.

What differs between HiveQL and Spark SQL?

Area	What the Spark docs say
Coverage	Spark SQL supports the vast majority of Hive features, including DISTRIBUTE BY, CLUSTER BY, SORT BY, window functions, UDFs, UDAFs and SerDes.
Views	When a view definition omits column aliases, Hive and Spark generate alias names differently. Specify aliases explicitly in Hive view definitions so Spark can read them.
Results	SQRT(n) with n < 0 returns null in Hive and NaN in Spark SQL. CAST(n AS TIMESTAMP) treats integral n as milliseconds in Hive and seconds in Spark SQL.
Unsupported	UNION type, unique join, Hadoop archive, and some Hive UDF APIs. Spark SQL also does not merge small result files.
Parallelism	Post-shuffle parallelism follows spark.sql.shuffle.partitions (default 200). Adaptive query execution is on by default and can coalesce shuffle partitions at runtime.

How do I convert Hive tables to Delta Lake or Iceberg?

Parquet-backed tables can convert to Delta Lake in place, which needs Delta Lake's SparkSession configuration for SQL commands. Iceberg has a test-copy procedure and a migration procedure, which need an Iceberg catalog configured in Spark:

CONVERT TO DELTA parquet.`<path-to-table>`;

CALL catalog_name.system.snapshot('db.sample', 'db.snap');
CALL catalog_name.system.migrate('db.sample');

CONVERT TO DELTA lists the files, creates a Delta Lake transaction log and infers the schema. For partitioned data you must also give the partition columns. Iceberg's snapshot creates a temporary copy without changing the source table. migrate replaces the table, retains the original as table_BACKUP_ by default, and fails for bucketed tables or partitions that are not Avro, Parquet or ORC.

What replaces Oozie?

Apache Oozie was a workflow scheduler for Hadoop jobs. It retired in February 2025 and moved to the Apache Attic in April 2025. Apache Airflow is an open-source platform for developing, scheduling and monitoring workflows defined in Python. Its Spark provider, apache-airflow-providers-apache-spark, includes SparkSubmitOperator, and the host of the spark_default connection can be yarn:

from datetime import datetime

from airflow.sdk import DAG
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator

with DAG(dag_id="hive_to_spark", start_date=datetime(2026, 1, 1), schedule="0 0 * * *"):
    SparkSubmitOperator(
        task_id="run_spark_job",
        application="/path/to/spark_job.py",
        conn_id="spark_default",
    )

How does Databricks help with this migration?

The Databricks migration guide recommends Delta Lake instead of Parquet or ORC, removing explicit SparkSession creation and terminal commands such as sys.exit(), and letting Databricks configure the driver and executors. Lakeflow Jobs orchestrate workloads on Databricks. Lakebridge, a migration toolkit that targets Databricks, lists Hive, Oozie, Pig and Sqoop as sources for its analyzer, which assesses existing code before migration.

FAQs

Is YARN required for Spark?

No. Spark runs on its standalone cluster manager, on YARN or on Kubernetes.

What happens to my Hive data?

It depends on the route. Reading through Hive support leaves the tables and their placement unchanged. CONVERT TO DELTA converts Parquet data in place, and Databricks advises that you avoid updating or appending files during the conversion and send all writes through Delta Lake afterward. Iceberg's snapshot leaves the source untouched.