How do I ingest unstructured data like PDFs and images into Databricks?

Summary

• Land the files in a Unity Catalog volume, read them as binary content with the binaryFile format, and parse them with the ai_parse_document SQL function. That gives you governed raw files plus structured, queryable output in Delta tables.
• ai_parse_document is generally available. It accepts PDF, JPG/JPEG, PNG, TIFF, DOC/DOCX and PPT/PPTX input and returns text, tables, figures, headers and bounding boxes as a VARIANT value.
• For files that keep arriving, Auto Loader supports the BINARYFILE format and reads new files incrementally, so you can pick up new documents as they arrive.
• After parsing, ai_extract (generally available) pulls named fields, and ai_prep_search (Beta) chunks the output for Databricks AI Search and retrieval-augmented generation.
• Managed file connectors in Lakeflow Connect ingest documents from enterprise file storage services, and the FILE type (Beta) stores file references directly in tables.

How do I ingest unstructured data like PDFs and images into Databricks?

Store the files in a Unity Catalog volume, load them as binary content, and run ai_parse_document to turn each document into structured elements you can query, extract from or index. The files stay governed in the volume, and the parsed output lands in Delta tables under the same permissions as the rest of your data.
The hard part isn't copying files. It's making their contents usable. A PDF in a bucket can't be filtered, joined or searched until something reads its layout, tables and figures.

What counts as unstructured data here?

Unstructured data is any file without a tabular schema: scanned and digital PDFs, Word and PowerPoint files, and images such as JPG, PNG and TIFF. In Databricks, these files are governed by Unity Catalog volumes, which govern non-tabular data of any format. Volumes can't be registered as tables; you access files by path and write any derived data to tables.

The ingestion pipeline, step by step

Step	What to use	What it does
Land files	Managed or external Unity Catalog volume	Governs access to the raw files. External volumes add governance to existing cloud storage paths.
Bring files in from file services	Managed file connectors in Lakeflow Connect	Handle authentication, incremental reads and retries for enterprise file storage services. Check each connector's release stage.
Read the bytes	read_files(..., format => 'binaryFile') or Auto Loader	Produces a binary content column per file. Auto Loader reads new files incrementally.
Parse	ai_parse_document	Extracts layout elements, text, HTML tables and figure descriptions.
Extract fields	ai_extract	Pulls named fields, such as invoice IDs or totals, from the parsed output.
Prepare for search	ai_prep_search (Beta)	Splits content into semantic chunks with document context for indexing.

How Databricks approaches unstructured ingestion

Parse with one SQL function. ai_parse_document reached general availability on April 16, 2026, per the release notes. Input must be binary data, which the binaryFile reader produces from files in a volume. The output schema (version 2.0) includes a document with pages and elements, plus error_status and metadata. Element types include text, table, figure, title, section_header, page_header, page_footer and footnote. Tables come back as HTML, and each element carries a confidence score and a bbox.
This example from the docs parses invoices in a volume and extracts fields:

WITH parsed_docs AS (
  SELECT
    path,
    ai_parse_document(
      content,
      MAP('version', '2.0')
    ) AS parsed_content
  FROM READ_FILES('/Volumes/finance/invoices/', format => 'binaryFile')
)
SELECT
  path,
  ai_extract(
    parsed_content,
    '["invoice_id", "vendor_name", "total_amount"]',
    MAP('instructions', 'These are vendor invoices.')
  ) AS invoice_data
FROM parsed_docs;

Handle images and figures. Image files go through the same function. Set imageOutputPath to save rendered page images to a volume for multimodal retrieval, and use descriptionElementTypes to generate AI descriptions of figures. Charts return as figure elements, with chart data as JSON when it can be read. The pageRange option limits parsing to selected pages.
Make it incremental. Auto Loader supports the BINARYFILE format and processes new files as they arrive, and the docs link a bundle example for incremental processing with ai_parse_document. The function also runs in notebooks, the SQL editor, jobs and Spark Declarative Pipelines (SDP).
Feed retrieval. ai_prep_search is in Beta. It turns ai_parse_document output, or plain text and markdown, into chunks enriched with document context such as titles, section headers and page references, and the docs show how to use the result as the source for a Databricks AI Search index. It needs Databricks Runtime 18.2 or above.
Reference files in tables. The FILE type is in Beta. It stores references to documents, images and audio in tables, either as managed copies or as references to files already in a volume.

Use cases

• Invoice processing: parse PDFs, then extract fields such as invoice IDs, vendor names and totals into tables.
• RAG chatbots: chunk policy and product documents so a chatbot can help users understand them.
• Scanned documents: parse image files such as PNG, JPG and TIFF into structured elements for downstream analytics.

FAQs

What file formats does ai_parse_document support?

PDF, JPG/JPEG, PNG, TIFF/TIF, DOC/DOCX and PPT/PPTX. The release notes state that documents are limited to a maximum of 500 pages and 100 MB.

What compute does ai_parse_document need?

Databricks Runtime 17.3 or above, or serverless with environment version 3 or above. The function is only available in some regions, so check AI function availability for your workspace.

Does my document data leave Databricks during parsing?

The docs state that document data is processed within the Databricks security perimeter. Databricks doesn't store the parameters passed into AI function calls, but it does retain metadata run details, such as the Databricks Runtime version used.

Can I validate parsing results without writing code?

Yes. The Document Parsing UI lets you validate and iterate on ai_parse_document results visually.