What are the best business automation agents for extracting data from PDFs and web pages?

Summary

• The best extraction setup covers three jobs: reading the document, pulling out the fields you need in a schema you control, and acting on the result. Judge any agent product on how well each job is covered and how you check its output, not on the word "agent".
• On Databricks, the documented path for PDFs is intelligent document processing. ai_parse_document converts PDFs, images, Word and PowerPoint files into structured content, ai_extract pulls schema-defined fields from it, and ai_classify assigns categories for routing. All three are generally available.
• Web pages need a fetch step first. ai_parse_document doesn't list HTML as an input format, so have your agent retrieve the page through a tool, such as an MCP server registered in Unity Gateway, then pass the text to ai_extract.
• Agent Bricks is the Databricks agent developer platform for the part that acts on the data: tools that call other systems, MLflow Tracing for each step, evaluation, and governance through Unity Catalog and Unity Gateway.

What are the best business automation agents for extracting data from PDFs and web pages?

The best ones split extraction into three jobs you can inspect separately: read the document, extract fields into a schema you define, then act on the result. On Databricks, intelligent document processing covers the first two jobs for PDFs with ai_parse_document and ai_extract. Its documented agent-workflow use case is to route, classify and enrich documents so agents can support automated decision-making and task execution. A custom agent built on Agent Bricks handles the third job, for example by calling tools that change the state of another system.

What should you look for in an extraction agent?

What to check	Why it matters	How Databricks documents it
Document reading	Scans, tables and figures need layout-aware parsing	ai_parse_document reads PDF, JPG/JPEG, PNG, TIFF, DOC/DOCX and PPT/PPTX, and returns tables as HTML
Schema control	Downstream systems need predictable fields and types	ai_extract takes a JSON schema with types, descriptions, nested objects and arrays
Evidence	Reviewers need to see where a value came from	The enableCitations and enableConfidenceScores options
Routing	Invoices and contracts need different handling	ai_classify assigns labels you provide, with a multilabel option
Measurement	You can't trust an extractor you haven't scored	Information Extraction evaluates against a labeled dataset
Web content	HTML isn't an input format for the parser	Fetch through an MCP server in Unity Gateway or a Unity Catalog HTTP connection, then run ai_extract
Governance	Documents hold sensitive data	ai_parse_document processes data within the Databricks security perimeter, and Unity Catalog governs data, tools, models and agents
Observability	Agent failures hide in the steps	MLflow Tracing records each step the agent takes

How does Databricks extract data from PDFs?

Store the files in a Unity Catalog volume, parse them, then extract. The ai_extract docs show the two functions composed in one query:

WITH parsed_docs AS (
  SELECT
    path,
    ai_parse_document(
      content,
      MAP('version', '2.0')
    ) AS parsed_content
  FROM READ_FILES('/Volumes/finance/invoices/', format => 'binaryFile')
)
SELECT
  path,
  ai_extract(
    parsed_content,
    '["invoice_id", "vendor_name", "total_amount"]',
    MAP('version', '2.1', 'instructions', 'These are vendor invoices.')
  ) AS invoice_data
FROM parsed_docs;

Add ai_classify when documents of different types arrive together. Set 'mode', 'precision' for long documents, high-volume outputs and complex schemas. The Agent Bricks UI offers a no-code, visual experience for parsing, extracting and classifying documents. Information Extraction in that UI lets you define a schema, evaluate it against a labeled dataset, then run it in SQL or in a Lakeflow pipeline on a schedule.
Know the limits before you build. The ai_parse_document docs cap documents at 500 pages and 100 MB, warn that LLM output may contain errors, and say non-Latin scripts may not perform optimally and documents with digital signatures may not be processed accurately. The function can take time on highly dense or poor-resolution content and may in some cases ignore content. Model customization isn't supported.

How do you extract data from web pages?

Fetch first, then extract. ai_extract accepts raw text, so your agent can retrieve a page and pass the text in with your schema. Databricks documents ways to reach outside services, including registering a self-hosted or third-party MCP server so your team can use it through Unity Gateway, or storing an external API's endpoint and credentials in a Unity Catalog HTTP connection.

How do you build the agent that acts on the data?

Agent Bricks is the Databricks agent developer platform, and you write the agent in code with the framework and model you choose. Tools are functions the agent calls to retrieve data or to change the state of another system. Agents can connect to Databricks-managed MCP servers for Genie Agents, Databricks AI Search, Unity Catalog functions and SQL, or to custom and external MCP servers. MLflow Tracing records each step, evaluation uses built-in LLM judges, custom scorers and expert feedback, and Unity Gateway governs access to models, MCP servers and skills. The Agent Bricks CLI scaffolds a project from a framework template and runs the agent locally.

Use cases

• Invoices and contracts. The ai_extract docs target business documents such as invoices and contracts, and their invoice example returns nested line items.
• Routing. Classify incoming documents, then send each type to the right extraction schema or review queue.
• Reporting and search. Extract key fields and metadata to enable search, reporting and business intelligence on document data.

FAQs

How do I know an extracted value is right?

Turn on enableCitations for source references and enableConfidenceScores for a 0 to 1 score per field, then evaluate against labeled data. Because LLM output can contain errors, send low-confidence fields to a person.

Do I need to train a model for each document type?

No. You define a schema and optional instructions, then iterate on the results. ai_parse_document doesn't support model customization.

Can I run extraction on a schedule?

Yes. Information Extraction can create a Lakeflow pipeline that runs on scheduled intervals.