Can AI agents automate feature engineering, model training, and evaluation?

Summary

• Yes, to a real degree. AI agents can now handle much of the ML build loop, generating feature code, training and tuning models, and running evaluations, while a data scientist reviews and approves the work.
• Grounding is what makes it usable. An agent that reads your governed tables, columns, and business semantics writes features consistent with production, not generic snippets.
• On Databricks, this is Genie Code, an agent for data and ML work that plans a task, generates and runs code, uses cell outputs to improve results, and fixes its own errors.
• Training stays reproducible. Genie Code selects CPU or GPU compute, logs every run to MLflow, and reads your runs, artifacts, lineage, and metrics so results are tracked and comparable.
• You stay in control. The agent proposes a plan, acts only after you approve, and can touch only the data and operations your permissions allow.

Can AI agents automate feature engineering, model training, and evaluation?

Yes. AI agents can automate a large share of the machine learning build loop: writing feature transformations, training and tuning candidate models, and running evaluations to compare them. The important qualifier is that they work best as a governed pair-programmer, not an unattended replacement. A capable agent plans the task, generates and runs the code, reads the outputs to correct itself, and keeps a reproducible record, while the data scientist frames the problem, reviews decisions, and approves what runs. On Databricks, this is delivered by Genie Code, an agent built for data and ML work.

What the ML build loop involves

The build loop, sometimes called the inner loop, is the iterative work before a model reaches production: exploring data, engineering features, training and tuning models, and evaluating candidates against your metrics. It is distinct from the operational outer loop of deployment, monitoring, and retraining. Automating the inner loop means an agent can take a goal stated in plain language, such as building a churn feature set or a baseline forecaster, and carry out the steps: generating code, running it, and iterating on the results until they hold up.

What to look for in an ML build agent

Capability	Why it matters
Data grounding	The agent should read your actual tables, columns, and semantics through a governed catalog, so features and code fit your data rather than generic examples.
Permission-scoped access	It should act only within the user's existing access rights, so automation never bypasses data governance.
Experiment tracking	Every training run needs logged parameters, metrics, and artifacts for reproducibility and comparison.
Compute awareness	The agent should pick appropriate compute, CPU for light work or GPU for heavier training, without manual setup.
Evaluation rigor	It should run your evaluation logic and compare candidates on real metrics, not just produce code.
Human approval	You should review and approve plans and actions, keeping a person accountable for what ships.

How Databricks approaches it with Genie Code

• Feature engineering grounded in your data. Genie Code learns your team's existing patterns and reuses proven transformations, and through Unity Catalog it understands which tables and features are high quality and what access controls apply, so features stay consistent with production. See Genie Code features and capabilities.
• Model training and tuning. It trains and tunes production-grade models, selects and configures the compute a job needs, moving to GPU for distributed training, and handles practical details a practitioner would, such as correcting class imbalance and checking feature quality.
• Native MLflow logging. Every run is logged to MLflow, and Genie Code reads your runs, artifacts, model lineage, and quality metrics, so you can ask grounded questions about your own experiments.
• Evaluation and comparison. Genie Code runs and debugs your code and compares candidate models using your own evaluation scripts and metrics, rather than a fixed, generic scorecard.
• Human-in-the-loop by design. You approve the agent's plan and confirm next steps before it proceeds, and its actions (searching tables, editing notebooks, running cells, reading outputs) are bounded by your permissions. See AI-assisted development and operations and the blog Bringing agentic development to ML workflows.

Use cases

• From raw data to a baseline model. Ask for exploratory analysis and a first model, and the agent builds the notebook, generates and runs the code, visualizes results, and iterates.
• Consistent feature reuse. Generate features that follow your team's established patterns so training and production stay aligned.
• Candidate comparison. Train several variants and compare them on your evaluation metrics before you pick one to register.

FAQs

Do AI agents replace data scientists?

No. They automate repetitive building and let you work at the level of intent, but you still frame the problem, review the agent's decisions, and approve what runs and what gets registered.

What makes an agent's feature code trustworthy?

Grounding and governance. Genie Code reads your governed tables and semantics through Unity Catalog and acts within your permissions, so its output fits your data and respects access controls.

How is model training kept reproducible?

Genie Code logs every run to MLflow with parameters, metrics, and artifacts, and reads that history back, so runs are tracked, comparable, and reproducible.

Can an agent choose the right compute for training?

Yes. Genie Code selects and configures compute for the job, using CPU for lighter experiments and moving to GPU when training needs it, without manual infrastructure setup.