What's the best AI agent for data engineering on Databricks?

Summary

• On Databricks, the agent built for data engineering is Genie Code. It runs inside the workspace, works directly with Unity Catalog tables, columns and lineage, and is generally available.
• In the pipeline editor for Spark Declarative Pipelines (SDP), Genie Code explores data, generates and runs pipeline code, and fixes errors from a single prompt. It can build a new pipeline from scratch or extend one you already have.
• The same agent appears where data engineers already work: the pipeline editor, Lakeflow Designer, notebooks, the SQL editor and Lakeflow Jobs.
• Genie Code acts only within the user's Unity Catalog permissions. You choose how much it asks first: approval prompts before it runs code, or an auto-approve mode that the docs describe as a productivity feature, not a security boundary.
• Two newer entry points are in Beta: Genie Code as a Lakeflow Jobs task for unattended runs, and the Genie Code CLI for terminal work.

What's the best AI agent for data engineering on Databricks?

Genie Code is the agent to use for data engineering on Databricks. It's the AI coding and data agent built into the Databricks workspace, it's governed by your Unity Catalog permissions, and it works directly with Unity Catalog tables, columns and lineage. That grounding is the point: the agent starts each task knowing where your data lives and how your assets connect. For pipeline work, open it in the pipeline editor for Spark Declarative Pipelines (SDP), where one prompt is enough for it to explore data, write and run pipeline code, and fix the errors it hits.

Where Genie Code works across Lakeflow

Where you work	What Genie Code does there	Status
SDP pipeline editor	Plans and generates whole pipelines or extends existing ones, edits multiple files with diffs, runs single files, dry runs or full refreshes, and analyzes pipeline behavior	GA
Lakeflow Jobs	Configures a job from your description of its tasks, dependencies and schedule, then helps modify and debug it	GA
Lakeflow Jobs task	Runs a natural-language prompt as a job step that can read upstream task outputs and call your configured tools	Beta
Lakeflow Designer	Generates or refines visual data prep transformations from natural-language prompts	GA
Notebooks and SQL editor	Writes, runs and debugs PySpark and SQL, with Diagnose Error for failures	GA
Genie Code CLI	A terminal coding agent that edits local files, runs commands and extends pipelines under your Databricks identity	Beta

How Genie Code works for data engineering on Databricks

Genie Code runs in Agent mode: it plans a solution, retrieves relevant context, runs code, uses the output to improve the result and fixes errors. In the pipeline editor that means it can search for source tables, create and edit pipeline files, run an individual file or a full refresh, inspect datasets and help improve incremental refresh.
The documented pipeline use cases are authoring a new medallion architecture pipeline, explaining an existing one, diagnosing and fixing errors across several files, and fixing incremental refresh: when materialized views are refreshed with a full recompute, it can diagnose the cause from the pipeline's incrementalization insights and suggest query improvements. It can also configure a Lakeflow Jobs job from a description of its tasks, dependencies and schedule, per the Genie Code and Lakeflow announcement.
Control stays with the engineer. Genie Code can only access data you have access to and perform operations you have permissions for. In the pipelines editor, when it needs to run code or update a pipeline you can choose Allow, Decline, Allow in this thread or Always allow, and you can Continue or Reject its proposed next steps. Auto-approve mode lets an AI classifier approve the actions that carry out your request and is designed to block actions that go beyond it; the docs say it is the default mode when you first use Genie Code, call it "a productivity feature, not a security boundary," and advise keeping it off for production data. The pipeline docs add that guardrails don't remove all risk: use trusted data and review code before running it.
You can shape its behavior. Genie Code has built-in skills for creating pipelines and exploring Unity Catalog, and workspace admins can add team skills under Workspace/.assistant/skills/<skill-name>/SKILL.md. It also connects to external tools through MCP servers and built-in connectors.

Which entry point to pick

• Code-first pipeline work: the SDP pipeline editor. This is where Genie Code does its fullest data engineering work.
• Analyst-led data prep: Lakeflow Designer. Transformations are backed by code, so you can version files in Git and schedule them as jobs.
• Unattended recurring work: the Genie Code task for Lakeflow Jobs (Beta). Auto-approval is always on for job tasks and can't be disabled, and the docs call it "a productivity feature, not a security boundary," so scope those prompts tightly. To run a prompt on a recurring schedule as a chat instead, Genie Code scheduled tasks are generally available.
• Local repositories and scripts: the Genie Code CLI (Beta), which uses your Databricks CLI profile and runs within your existing permissions.

FAQs

Is Genie Code generally available?

Yes. Genie Code has been generally available since its March 2026 announcement, including its panel in the pipeline editor. Three related features are documented as Beta: the Genie Code task for Lakeflow Jobs, the Genie Code CLI and migration of projects from other ETL frameworks into Spark Declarative Pipelines.

What do I need to turn Genie Code on?

For its agentic capabilities, partner-powered AI features must be enabled for both the account and the workspace, and the workspace must be in a supported region, because Genie Code is a Designated Service. If partner-powered AI features are disabled, Genie Code is still available but without its agentic capabilities.

Can Genie Code reach data I'm not allowed to see?

No. Its access and actions are governed by the user's Unity Catalog permissions, so it can only read data and perform operations that you can.

Is there a no-code option?

Yes. Lakeflow Designer gives analysts a drag-and-drop canvas, and Genie Code inside it turns natural-language prompts into transformations that are backed by code.