How do you use RAG over structured and unstructured enterprise data?

Summary

• Enterprise RAG requires combining text-to-SQL for structured data with vector search for unstructured documents, using query routing and reranking to merge results before generation.
• Best practices include hybrid search, metadata filtering, domain-tuned embeddings, centralized access controls, and continuous evaluation loops to maintain retrieval quality.
• Agent Bricks on the Databricks Platform provides a unified control plane to build governed RAG agents, with the Knowledge Assistant Brick automating ingestion and embedding while Unity Catalog enforces consistent permissions and lineage tracking.

RAG over structured and unstructured enterprise data

Enterprise knowledge lives in many places: SQL databases, data warehouses, PDFs, spreadsheets, support tickets, and internal wikis. Building a single RAG pipeline that handles all these formats reliably is one of the hardest problems in enterprise AI.
Most retrieval systems were designed for a simpler world, a single index, a uniform schema, and similarity search as the primary signal. Enterprise knowledge spans many systems with different structures, metadata conventions, and expectations around freshness and authority.

Why enterprise RAG fails without business context

Conventional RAG methods struggle with enterprise datasets for several overlapping reasons:

• Format fragmentation: Structured tables, PDFs, images, and emails each require different parsing and embedding strategies.
• Missing semantics: An LLM cannot distinguish between a "revenue" column in finance and a "revenue" field in sales without business context.
• Governance gaps: Data spans multiple systems with different permission models, risking exposure of sensitive content.
• No unified framework: No widely adopted standard integrates hybrid retrieval, structured data handling, metadata filtering, reranking, and interactive refinement.

According to IDC, 90% of the data generated by organizations is unstructured, including documents, emails, images, and videos. Any retrieval system limited to structured queries alone misses the vast majority of organizational knowledge.

Architectural patterns for mixed-format enterprise RAG

A robust enterprise RAG pipeline typically combines multiple retrieval paths rather than forcing all data through a single index.

Text-to-SQL for structured data

Text-to-SQL translates natural language questions into executable SQL at query time. It grounds the LLM in meaning encoded within database structures, metadata, and domain documentation. Key practices include:

• Provide the LLM with schema descriptions, column-level comments, and sample rows.
• Use few-shot prompting with validated query examples.
• Validate generated SQL against the schema before execution.

Vector search for unstructured documents

Unstructured content requires parsing, chunking, embedding, and indexing before retrieval:

• Layout-aware parsing preserves headings, tables, and reading order.
• Chunking strategies should balance granularity with context, too small loses meaning, too large dilutes relevance.
• Metadata preservation (source, date, author, permissions) enables filtering at query time.

Query routing and result merging

Route each query to the appropriate retrieval path, SQL for structured questions, vector search for document questions, then merge and rerank results before generation. A router can be rule-based, classifier-based, or LLM-driven.

Best practices for enterprise-scale RAG

Practice	Why it matters
Hybrid search (dense + sparse)	Combines semantic similarity with keyword precision
Metadata filtering	Narrows retrieval to relevant sources, time ranges, or departments
Reranking	Improves precision after initial retrieval
Domain-tuned embeddings	Captures terminology specific to your industry
Centralized access controls	Ensures users only retrieve content they are authorized to see
Evaluation loops	Measures retrieval quality and generation accuracy continuously

How Agent Bricks addresses enterprise RAG challenges

Agent Bricks (Mosaic AI Agent Framework) is the unified control plane to build, run, and govern AI agents across any model, provider, or framework. Built natively into the Databricks Platform, it grounds agents in semantic knowledge graphs that understand your business data, enabling contextual reasoning over both structured tables and unstructured documents.

• Open and governed: Build with any AI model, OpenAI, Anthropic, Gemini, Llama, while maintaining granular access controls, lineage tracking, and policy enforcement through Unity Catalog.
• Knowledge Assistant Brick: A no-code RAG agent that handles ingestion, embedding, retrieval, and inference with citations. It removes the need to manually tune chunking and embedding pipelines.
• Self-improving quality: Agent Bricks builds benchmarks using your own data and tasks, evaluates every output, and improves performance through prompt optimization, fine-tuning, RLHF, and human feedback.

Unity Catalog provides the governance layer, ensuring agents accessing both structured tables and unstructured documents inherit consistent permissions and lineage tracking.

FAQs

How do you implement RAG over structured data like SQL databases and data warehouses?

Convert structured records into text representations or use text-to-SQL to translate natural language queries into executable SQL at query time. Provide schema metadata and validated examples to improve accuracy.

What are the best techniques for chunking and embedding unstructured enterprise documents for RAG?

Use layout-aware parsing that preserves headings, tables, and reading order before chunking. Preserve metadata rather than flattening everything into plain text. The Knowledge Assistant Brick automates ingestion and embedding.

How can a single RAG pipeline handle both structured tables and unstructured text simultaneously?

Route queries to specialized retrieval paths, text-to-SQL for structured data and vector search for documents, then merge and rerank results before generation.

What vector database architectures work best for enterprise-scale RAG deployments?

Choose architectures that support hybrid search (dense and sparse), metadata filtering, and role-based access control. Scalability, low-latency retrieval, and integration with your governance layer are critical selection criteria.

How do you convert structured data into a format suitable for retrieval-augmented generation?

Serialize rows into natural language sentences or key-value descriptions, preserving column names and context. Alternatively, use text-to-SQL so the LLM queries structured data directly without conversion.

What are the challenges of building RAG systems over enterprise data with mixed formats?

Mixed formats require different parsers, chunking strategies, and embedding approaches. Governance becomes complex when data spans multiple systems with different permission models.

How does text-to-SQL integrate with RAG to answer questions over structured enterprise data?

Text-to-SQL translates natural language into executable SQL, grounding responses in actual database values. Combining it with document retrieval gives the LLM both precise data and supporting context.

What are best practices for metadata filtering and hybrid search in enterprise RAG systems?

Tag documents with source, date, department, and access-level metadata at ingestion time. Combine dense vector search with sparse keyword matching, then apply metadata filters to narrow results before reranking.

How do you handle data governance and access control in enterprise RAG pipelines?

Enforce permissions at the data layer so agents only retrieve content the requesting user is authorized to see. Unity Catalog provides centralized access control, lineage tracking, and sensitive-data classification across all data types.

What embedding models and retrieval strategies are most effective for enterprise knowledge bases?

Use domain-tuned embedding models and hybrid retrieval combining dense vector search with sparse keyword matching. Continuous evaluation against your own benchmarks helps identify when retrieval quality degrades.

Build governed RAG agents that understand your enterprise data

Enterprise RAG demands contextual reasoning grounded in business semantics, unified governance, and continuous quality improvement. Agent Bricks, built natively into the Databricks Platform with Unity Catalog, delivers these capabilities in a single control plane, helping you build, run, and govern AI agents that reason accurately over structured and unstructured enterprise data. Explore Agent Bricks to get started.