What is RAG in AI?

Summary

• RAG (retrieval-augmented generation) combines a large language model with retrieval from an outside knowledge base, so the model answers with supporting data fetched at request time.
• The Databricks docs describe three steps: retrieval from a source such as a vector store, keyword search, or SQL database; augmentation of the user's request with the supporting data; and generation by the LLM.
• Documented benefits are proprietary knowledge, up-to-date information, source citations, and access control in the retrieval step.
• A typical RAG application has a data pipeline, a RAG chain, evaluation and monitoring, and governance.
• On Databricks, AI Search covers retrieval, MLflow covers tracing and evaluation, and Unity Catalog governs the assets.

What is RAG in AI?

Retrieval-augmented generation (RAG) is an AI architecture pattern that combines a large language model (LLM) with data retrieval. Instead of answering only from what the model learned in training, a RAG system looks up supporting data when the user asks, adds it to the request, and passes the result to the LLM to generate a response. Databricks describes the goal as more accurate, up-to-date, and contextually relevant responses.

How does RAG work?

The Databricks docs describe three functions:

• Retrieval. The user's request is used to query an outside knowledge base such as a vector store, keyword search, or SQL database.
• Augmentation. The supporting data is combined with the user's request, often using a template with additional formatting and instructions.
• Generation. The prompt is passed to the LLM to generate a response.

For unstructured documents, the data pipeline behind retrieval ingests and parses the content, splits it into chunks, embeds the chunks, and indexes them. The docs name fixed-size, paragraph-based, format-specific, and semantic chunking. Smaller chunks may focus on specific details but lose surrounding context, and larger chunks may capture more context but can include irrelevant information.

What are the benefits of RAG?

Benefit	What the Databricks docs say
Proprietary knowledge	RAG can incorporate proprietary information the LLM was not trained on, such as memos, emails, and documents
Up-to-date information	A RAG application can supply the LLM with information from an updated knowledge base
Source citations	RAG enables LLMs to cite specific sources, so users can verify the factual accuracy of responses
Access control	The retrieval step can be designed to selectively retrieve personal or proprietary information based on user credentials

Answer quality still depends on retrieval. Databricks notes that agent performance depends on retrieving the most relevant information for a query, so evaluation covers retrieval as well as the generated response.

How does Databricks approach RAG?

Databricks describes an end-to-end platform for RAG development, with integrated data pipelines, scalable AI Search, model serving and orchestration tools, evaluation, monitoring for deployed applications, and built-in governance and security. The table maps the stages to documented capabilities.

Stage	Databricks capability
Parse documents	The ai_parse_document SQL function parses structured content from unstructured documents such as PDFs, images, Word documents, and PowerPoint files
Index and retrieve	Databricks AI Search indexes can sync with a source Delta table, use Databricks-computed or self-managed embeddings, and support hybrid keyword-similarity search, metadata filters, and reranking
Build the agent	Agent Bricks is the Databricks agent developer platform, and its agents can connect to Databricks-managed MCP servers for AI Search. Genie Agents is a low-code way to build agents that answer questions over tables and documents
Trace and evaluate	MLflow Tracing records each step an agent takes, in development and in production. Built-in judges include RetrievalGroundedness, RetrievalRelevance, and RetrievalSufficiency
Govern	AI Search indexes appear in and are governed by Unity Catalog

One limit matters for access control. AI Search does not support row and column level permissions, and the docs suggest implementing application level ACLs with the filter API.
Managed agent memory (Beta, backed by Lakebase) is a separate capability from RAG retrieval. It gives agents durable, long-term memory that persists across conversations.

FAQs

How does RAG differ from fine-tuning?

Fine-tuning trains a model further on your data. RAG does not retrain the model. It supplies supporting data in the prompt at request time. The two can be combined: the Databricks RAG chain guidance suggests considering fine-tuning smaller models for specific sub-tasks within the chain, such as query understanding.

How is RAG different from a search engine?

A search system returns matching results for you to read. A RAG system passes retrieved data to an LLM, which writes a response and can cite specific sources.

How do you know if retrieval found the right information?

Check retrieval separately from generation. RetrievalRelevance asks whether the retrieved context is directly relevant to the request. RetrievalSufficiency asks whether the context provides all necessary information to include the ground truth facts, and it needs ground truth. RetrievalGroundedness asks whether the response is grounded in the context.

Can RAG use structured data?

Yes. The Databricks docs describe unstructured data such as PDFs, documents, wikis, images, and videos, and structured data such as customer records, transaction data, and application APIs. The retrieval step can query a vector store, keyword search, or SQL database.

Is agent memory the same as RAG?

No. RAG retrieves from a knowledge base at request time. Managed agent memory stores facts and preferences that an agent recalls in later conversations. It provides long-term memory only, and short-term conversation history uses managed agent sessions.