How do I get started building AI and ML on Databricks?

Summary

• Databricks gives you an integrated, end-to-end environment with managed services for developing and deploying AI and ML, unifying the full lifecycle from data preparation to production and monitoring on the Data Intelligence Platform and governed by Unity Catalog.
• New users start with a quickstart to build and deploy a first model, then work in collaborative notebooks (Python, R, Scala, SQL) backed by Databricks Runtime for Machine Learning with prebuilt libraries and GPU support.
• For generative AI, use the AI Playground to test and compare models, Foundation Model APIs to build with state-of-the-art models, and Agent Bricks to build agents low-code.
• Core building blocks include AutoML, the Feature Store, MLflow for experiment tracking, model registry, and tracing, and Model Serving to deploy any model as a governed REST API.
• Everything is governed by Unity Catalog, with one copy of your data and shared definitions across every tool.

How do I get started building AI and ML on Databricks?

Databricks provides an integrated end-to-end environment with managed services for developing and deploying AI and machine learning on the Data Intelligence Platform. Whether you are training classic ML models or building generative AI applications, you develop, deploy, govern, and monitor them in one place, with Unity Catalog governing the data, features, and models throughout. Databricks Mosaic AI is the data-native, collaborative solution for this full ML lifecycle.

Why Databricks for AI and ML

• One integrated platform for the full lifecycle. The Data Intelligence Platform unifies data preparation, model development, deployment, and monitoring, so your teams do not stitch together separate tools. The AI stack, including MLflow, AI Search, Model Serving, and Agent Bricks, works together on one platform with unified governance.
• Collaborative notebooks and an ML runtime. Develop in notebooks that support Python, R, Scala, and SQL. Databricks Runtime for Machine Learning automates compute creation with prebuilt ML and deep learning libraries and pre-configured GPU support and drivers.
• Faster model building with AutoML. AutoML automatically generates models from your data, handling data preparation, model selection, and tuning to accelerate the path to production.
• Consistent features with the Feature Store. The Feature Store lets you share and discover features across your organization and ensures the same feature computation is used for both training and inference.
• Lifecycle management with MLflow. MLflow manages the ML lifecycle through experiment tracking, a model registry, and tracing, and its experiments collect the runs you use to develop ML models, LLM applications, and agents.
• Deploy anything with Model Serving. Model Serving provides a unified interface to deploy, govern, and query models, exposing each model as a REST API. It supports custom models, foundation models, and external third-party models.
• Build generative AI. Use Foundation Model APIs for pay-per-token or provisioned-throughput access to foundation models, the AI Playground to test, prompt, and compare models side by side, and Agent Bricks to build chatbots, information-extraction agents, and multi-agent systems low-code. AI Functions such as ai_query(), ai_extract(), and ai_parse_document() apply AI directly to data with SQL.
• Governed by Unity Catalog. Unity Catalog provides centralized access control, lineage, and cross-workspace discovery. Models registered in Unity Catalog inherit fine-grained governance alongside your tables and features.

Getting started

1. Follow Get started: Build your first machine learning model on Databricks to build and deploy a simple model.
2. Create a notebook in your workspace; it comes with a default MLflow experiment that serves as the container for your work.
3. Open the AI Playground to experiment with and evaluate models and agents using your own prompts.
4. Review Concepts: AI and machine learning on Databricks and the machine learning concepts to understand the fundamentals.
5. Platform teams: configure Unity Catalog for centralized governance of data, features, and models before scaling out.

FAQs

What do I need to start building ML on Databricks?

A Databricks workspace and a notebook. Databricks Runtime for Machine Learning provides the prebuilt libraries and GPU support, and the quickstart walks you through building and deploying your first model.

How do I get started with generative AI on Databricks?

Start in the AI Playground to test and compare models, build with the Foundation Model APIs, and use Agent Bricks to create chatbots, extraction agents, and multi-agent systems low-code.

What is Databricks Mosaic AI?

Mosaic AI is the data-native, collaborative solution for the full ML lifecycle on the Databricks Data Intelligence Platform, bringing governance, observability, and operational tooling together with popular open-source frameworks.

How are AI and ML governed on Databricks?

Unity Catalog governs data, features, and models with centralized access control, lineage, and discovery, so models and their data share one governance model.