What are the best data engineering skills to use for my agents to maintain my pipelines?

Summary

• Agent skills are task-specific instruction files that coding agents load for a job. For pipeline maintenance, start with databricks-core, databricks-pipelines, databricks-jobs, databricks-dabs, and databricks-unity-catalog, then add ingestion, streaming, or migration skills your pipelines need.
• Skills give an agent knowledge, while managed MCP servers give it tools. Installing skills does not configure MCP servers, so connect the servers your agent needs.
• Install with databricks aitools install. In the workspace, Genie Code has built-in skills and loads custom skills from .assistant/skills/ folders.
• Agents act with the permissions you give them. Use a dedicated identity, review diffs, and approve runs before anything changes.

What are the best data engineering skills to use for my agents to maintain my pipelines?

For pipeline maintenance on Databricks, load the skills that match the work. Use databricks-pipelines for Lakeflow Pipelines code, databricks-jobs for orchestration, databricks-dabs for deploying changes as code, databricks-unity-catalog for lineage, audit, and billing system tables, and databricks-core for the CLI and authentication. Add ingestion, streaming, or migration skills when your pipelines use them. All of these come from the officially maintained Databricks agent skills repository.

What is an agent skill?

Agent skills are task-specific instruction files that AI coding assistants load to perform Databricks development tasks. They follow the open Agent Skills standard: a directory with a required SKILL.md file whose frontmatter has a name and a description. Assistants discover and load relevant skills automatically based on the task.

Which Databricks skills fit pipeline maintenance?

Maintenance work	Skill	What the repository says it covers
Pipeline code fixes and tuning	databricks-pipelines	Lakeflow Pipelines for batch and streaming
Orchestration changes	databricks-jobs	Lakeflow Jobs task types, triggers, schedules, notifications
Shipping changes from version control	databricks-dabs	Declarative Automation Bundles for deploying and managing resources
CLI, authentication, data exploration	databricks-core	The parent skill for all product skills
Lineage, audit, and cost questions	databricks-unity-catalog	Unity Catalog system tables for lineage, audit, and billing
Streaming code	databricks-spark-structured-streaming	Spark Structured Streaming patterns
Ingestion	databricks-lakeflow-connect, databricks-zerobus-ingest	Managed connectors; Zerobus streaming ingest patterns
Compute upgrades	databricks-serverless-migration	Moving classic-compute workloads to serverless

The plugin versions for some agents also add a read-only doctor command that checks CLI version, authentication, workspace reachability, compute, and recent job failures.

How do I install and use them?

databricks aitools install --skills databricks-pipelines,databricks-jobs,databricks-dabs
databricks aitools list

The CLI detects supported coding agents and installs the skills for them. Add --scope project to install into the current project, and use databricks aitools update and databricks aitools uninstall to manage them. Skills teach the agent how to work; MCP servers let it do the work, and installing skills does not configure MCP servers for you.
Inside the workspace, Genie Code ships with built-in skills, including pipeline creation. Workspace admins can add custom skills as Workspace/.assistant/skills/<skill-name>/SKILL.md to enforce team standards, and Genie Code loads them when relevant or when you @ mention them. Back the folder with a Databricks Git folder to version it.

What should a custom maintenance skill contain?

Keep each skill focused on one workflow, with a clear name and description, step-by-step instructions, examples, and edge cases. Keep guidance in Markdown and repeatable actions in scripts. A minimal triage skill built from the pipeline docs looks like this:

---
name: pipeline-triage
description: Triage a failed Lakeflow Pipelines update. Use when an update fails or a table looks stale.
---

# Steps
1. Work backward from the failed node on the pipeline graph to its error in the event log.
2. Re-run only what failed. Ask a human before any full refresh.
3. Report the root cause, the fix, and the downstream tables affected.

# Edge cases
- Manually triggered updates do not retry, while scheduled updates retry recoverable failures.

The pipeline docs list other degradation signals worth encoding: rising update duration, autoscaling hitting its ceiling, and materialized views falling back to full recomputes.

What guardrails should agents have?

Genie Code in Agent mode acts within the user's permissions, asks for approval before it runs code or updates a pipeline, and shows diffs. The docs still warn that there is risk: use it with data you trust and review code before running it. The pipeline docs recommend running pipelines as a dedicated service principal rather than a personal account, with credentials in a secret scope.

FAQs

Does a skill let my agent read workspace data?

No. A skill carries knowledge. Reading live data takes a tool, such as a managed MCP server hosted by Databricks and governed by Unity Catalog.

Are experimental skills safe to rely on?

They are not officially supported, and databricks aitools install skips them unless you pass --experimental.

Can agents replace the on-call data engineer?

Not by themselves. Genie Code today targets authoring pipelines rather than diagnosing production runs, so keep monitoring, notifications, and human approval in place.