Can AI agents help migrate legacy data workflows to modern platforms?

Summary

• Yes. AI agents can take on the heaviest part of a migration, converting legacy SQL and ETL logic, and they validate and repair their own output before an engineer reviews it.
• On Databricks, the agentic code converter uses Genie Code to convert existing SQL from common warehouse dialects into Databricks SQL or, in Beta, Databricks notebooks, with a status for every file in a migration project.
• Genie Code can also migrate an existing data transformation project into a pipeline. This capability is in Beta: Genie Code reads the source, builds an intermediate representation, converts it, and repairs the result in a loop.
• Lakebridge, a Databricks Labs toolkit, covers the steps around conversion: assessing source workloads up front and reconciling migrated data against the source afterward.
• Treat converted code as a draft until it's proven. Databricks advises running the original and converted queries against equivalent test data and comparing the results.

Can AI agents help migrate legacy data workflows to modern platforms?

Yes. AI agents are well suited to the most labor-intensive part of a migration: converting large volumes of legacy SQL and ETL logic into code that runs on the new platform. A useful migration agent doesn't stop at translation. It validates the output, fixes what fails in a loop, and marks what still needs a person. Planning, ordering dependent workloads, and proving that migrated data matches the source remain engineering work, and the best tooling puts those steps right next to the agent.

Why is migrating legacy data workflows hard?

Legacy workflows are full of dialect-specific SQL, proprietary data types, and scripts that depend on each other and on shared tables. Converted code can be syntactically valid and still return different results, so a migration is only done when the output is semantically equivalent. Dependencies also decide the order: interdependent files and tables have to move together.

What to look for in an AI migration agent

Capability	Why it matters
Assessment before conversion	Code size, complexity and lineage shape the plan and the order of work.
Validation and repair loop	The agent should fix its own errors before a reviewer sees the file.
Per-file status and diffs	Reviewers can focus on files that failed or need attention.
Custom conversion rules	Organization-specific types and patterns should be encoded once and reused.
Choice of target	Some logic belongs in SQL, some in notebooks or declarative pipelines.
Data reconciliation	Only a source-to-target comparison proves the migration is correct.

How Databricks approaches AI-assisted migration

Phase	Databricks tool	Stage
Assess	Lakebridge Analyzer and Profiler	Databricks Labs project
Convert SQL	Agentic code converter in Genie Code	Notebooks target in Beta
Migrate ETL projects	Genie Code ETL-project migration, a Lakebridge capability	Beta
Reconcile	Lakebridge Reconcile	Databricks Labs project

Assess the source. Lakebridge profiles existing SQL workloads and analyzes the code, and where needed its orchestration, to report size, complexity and potential issues before you start. Lakebridge is provided by Databricks Labs as-is, without formal support SLAs.
Convert SQL with an agent. The agentic code converter uses Genie Code to convert existing SQL from several common warehouse dialects into code that runs on Databricks. You create a migration project, choose a source dialect and a target framework (SQL for ANSI-compliant Databricks SQL, or Notebooks, in Beta, for notebooks that use the DataFrame API), and add files. When you run a conversion, Genie Code invokes the migration skill and deploys subagents that convert files concurrently, validating each result and retrying sections that fail. Each file gets a status (converted, needs review, failed, or in progress), and SQL results open as a side-by-side diff in the SQL editor. If you upload the Lakebridge Analyzer output, the project also shows a complexity assessment and a lineage graph of files and tables that must move together. For a single file or folder, use the /migrate command in a Genie Code conversation.
Encode your own rules. A migration skill is a Genie Code skill. The docs' example is a rule to "Always convert the custom type MYDATE to TIMESTAMP_NTZ," which Genie Code then applies on every conversion in the workspace.
Migrate ETL projects into pipelines (Beta). Genie Code can migrate an existing data transformation project into a pipeline built on Spark Declarative Pipelines (SDP). Upload the project, create an empty pipeline, and prompt Genie Code with the project's path. It reads the source to understand models, transformations and dependencies, asks whether to generate SQL or Python, builds an intermediate representation of the pipeline's logic, then converts, validates and repairs until the pipeline is correct. This capability is part of Lakebridge and supports a limited set of source ETL frameworks today, with more planned.
Reconcile and test. Lakebridge Reconcile compares the source system against the Databricks target and reports discrepancies at the row, column and schema level. The converter docs also suggest converting a small batch first, running the output to confirm it executes, including scripts with dialect-specific constructs, and comparing original and converted query results on equivalent test data.

FAQs

Can AI fully automate a data migration?

No. Agents can convert, validate and repair code at scale, but engineers still plan the order of work, review flagged files, and confirm the migrated pipeline's results match the original before production.

How do I know AI-converted code is correct?

Check that it runs, then check that it means the same thing. Run original and converted queries against equivalent test data and compare results, and use Lakebridge Reconcile to compare migrated data with the source.

Can I teach the migration agent my organization's conventions?

Yes. Create a custom migration skill, from Genie Code settings or by asking in chat, and Genie Code applies its rules on future conversions.

Do I need to migrate everything at once?

No. Lineage from the Lakebridge Analyzer shows which files and tables depend on each other, so you can move interdependent groups together and convert in small batches.