How to run LLM inference directly on data in a lakehouse

Summary

• Run LLM inference directly on lakehouse data using SQL AI functions like ai_query() and ai_classify() without moving data or changing your governance model.
• Execute inference at scale using SQL, Python notebooks, or scheduled Lakeflow Pipelines with automatic fault tolerance and retries.
• Apply any AI model (foundation models, fine-tuned, or custom) across your lakehouse data efficiently while keeping data governed by Unity Catalog.
• Reduce engineering complexity by running preprocessing, inference, and postprocessing on one platform instead of building multi-system pipelines.
• Use structured output formats and response validation to ensure consistent, reliable inference results.

How to run LLM inference directly on data in a lakehouse

You can run LLM inference directly on lakehouse data using SQL AI functions without copying data to external systems or bypassing governance. The simplest approach is to use built-in SQL functions like ai_query() and ai_classify() to apply inference directly to tables in your lakehouse. The infrastructure includes built-in fault tolerance and retries at scale. You can execute inference interactively in the SQL editor, from Python notebooks, or as part of scheduled pipelines using Lakeflow Pipelines.

What are SQL AI functions?

SQL AI functions are built-in SQL operations that let you invoke LLM capabilities directly in queries. Available functions include ai_query() for general LLM tasks, ai_classify() for categorization, ai_extract() for structured data extraction, ai_summarize() for text summarization, ai_translate() for language translation, and ai_similarity() for semantic similarity matching. These functions accept your data as input, send it to the model endpoint, and return results back into your query context, so the entire operation stays within your governed data environment.

How to run LLM inference on lakehouse data

Approach	Best for	Example
SQL queries	Ad-hoc exploration, simple inference	SELECT ai_classify(text, 'sentiment') FROM articles
Python notebooks	Development and testing	Use spark.sql() or DataFrame APIs with AI functions
Lakeflow Pipelines	Scheduled batch inference at scale	Materialized view with AI function, runs on schedule
Workflows + SQL	Production pipelines	Multi-step job with inference as one task

How Databricks approaches LLM inference on lakehouse data

Execute inference using SQL AI functions. Create a simple SQL query that applies the AI function to your data. For example, to extract structured data from text:

CREATE OR REFRESH MATERIALIZED VIEW articles_categorized AS
SELECT
  article_id,
  title,
  content,
  ai_extract(
    'system.ai.meta-llama-3-3-70b-instruct',
    'Extract the main topic, sentiment, and key entities from this article: ' || content
  ) AS extracted_data
FROM articles_raw;

Scale inference using Lakeflow Pipelines. For larger datasets, materialize the inference results in Delta Lake using a scheduled pipeline. Lakeflow Pipelines manages retries, fault tolerance, and incremental updates automatically.
Use structured output formats. Specify response formats with JSON schema to enforce consistent output structure. This ensures downstream processes receive predictable, validated results:

SELECT
  ai_query(
    'system.ai.meta-llama-3-3-70b-instruct',
    'Categorize this product: ' || product_description,
    responseFormat => '{
      "type": "json_schema",
      "json_schema": {
        "name": "product_category",
        "schema": {
          "type": "object",
          "properties": {
            "category": {"type": "string"},
            "confidence": {"type": "number"}
          }
        },
        "strict": true
      }
    }'
  ) AS result
FROM products;

Deploy any AI model. Use any foundation model available through Databricks' Unity Gateway, or deploy your own fine-tuned or custom models. Meta Llama 3.3 70B Instruct and other frontier models are available; you choose which model fits your use case.
Govern inference with Unity Catalog. Because inference runs directly on lakehouse data, all access is governed by the same Unity Catalog policies you use for analytics. Row filters and column masks apply, audit logs record every inference query, and sensitive data stays protected throughout.

Use cases

Content categorization and tagging. Automatically classify documents, tickets, or messages into predefined categories using ai_classify(), then store results in new columns for downstream analytics and applications.
Information extraction. Extract structured data like entities, dates, amounts, and relationships from unstructured text (articles, emails, contracts, logs) and materialize it as queryable tables.
Sentiment and intent analysis. Analyze customer feedback, survey responses, or support transcripts in bulk without exporting data, then join results with customer data for segmented insights.
Text summarization. Summarize long documents, articles, or conversations at scale, keeping summaries in your lakehouse for faster retrieval and downstream applications.
Data quality and anomaly detection. Use inference to validate data quality, detect outliers, or flag suspicious patterns in logs or transactional data directly in your pipelines.

FAQs

Do I have to move data to a separate inference service?

No. Inference runs directly on data in your lakehouse. Data stays governed by Unity Catalog, so no data movement, no governance bypass, and no external dependencies.

What models can I use?

You can use foundation models available through Unity Gateway (like Meta Llama), or deploy your own fine-tuned or custom models to your Databricks workspace. You choose the model that fits your use case and performance requirements.

How do I handle large-scale inference?

Use Lakeflow Pipelines to schedule batch inference as a materialized view. The platform manages retries and fault tolerance automatically. You can also use Lakeflow Jobs to orchestrate multi-step inference pipelines.

Can I use inference results in real-time applications?

Yes. Store inference results in Delta Lake and serve them through Model Serving or Databricks Apps. For lower-latency applications, you can also call the inference API directly from your application.

How are inference queries audited?

Unity Catalog captures centralized audit logs of every query, including AI function invocations. These logs show who ran what query, when, and which data was accessed, so you have full visibility for compliance and troubleshooting.