How to Ingest SharePoint Documents into Databricks with Lakeflow Connect and AI Parse Document

Summary

Managed ingestion: Lakeflow Connect provides fully managed, incremental ingestion from SharePoint, Google Drive, email and SFTP into Unity Catalog.
PDF parsing: one SQL call converts PDFs into structured tables with bounding boxes and confidence scores.
Incremental processing: only new or changed documents are processed, reducing costs at scale.

How to Ingest SharePoint Documents into Databricks with Lakeflow Connect and AI Parse Document

Talk by Jason Ping, Product Manager, Databricks
Intelligent document processing on Databricks combines two pieces: Lakeflow Connect for fully managed, incremental ingestion of unstructured files from sources such as SharePoint, Google Drive, email and SFTP, and AI Parse Document to turn those PDFs into structured, queryable tables. This video walks the whole flow in minutes, including a five-step SharePoint connector setup, what lands in Delta as file metadata plus binary content, and parsing PDFs with bounding boxes and confidence scores under Unity Catalog governance.
Jason Ping, Product Manager at Databricks, explains why documents scattered across SharePoint, Google Drive, email and SFTP are hard to pipeline at scale, then introduces Lakeflow Connect as Databricks' managed ingestion platform that handles schema changes and lineage under Unity Catalog governance. He walks through the five-step SharePoint connector wizard, including a Unity Catalog connection with OAuth, and shows how files land in Delta with metadata and binary content. One line of SQL calls AI Parse Document, which returns bounding boxes, content and confidence scores. Because pipelines are incremental, only new or changed files are processed. He then builds a Genie space on the parsed table and previews upcoming sources such as OneDrive, Box and Dropbox.

Chapters

FAQs

How do I get SharePoint documents into Databricks?

Lakeflow Connect is a managed ingestion platform that handles the five-step SharePoint connector setup, including OAuth. Files land in Delta with metadata and binary content, governed by Unity Catalog.

How do I turn PDFs into structured data?

One line of SQL calls AI Parse Document, which returns bounding boxes, content and confidence scores. The parsed data becomes queryable tables ready for dashboards, Genie spaces or AI agents.

What is incremental document processing?

Lakeflow Connect only processes new or changed files, not everything in SharePoint every time. This makes ingestion cost-effective when you have millions of documents.

What document sources does Lakeflow Connect support?

Currently SharePoint, Google Drive, email and SFTP. The roadmap includes OneDrive, Box, Dropbox and file-level permissions so you can respect folder security.

Full transcript

`00:00` Introduction

Thank you to everyone who took the time to come to this session. I know this is the last day of the Data and AI Summit. I really appreciate you coming. This session is titled " Unstructured Data Ingestion with Lakeflow Connect." Brief introduction: My name is Jason. I'm one of the product managers here at Databricks. I am working on Lakeflow Connect . I'm happy to show you what we've prepared. Standard forward-looking statement .
And please don't forget to fill out the survey. This really helps us understand what quality and materials we will be able to provide in the coming years. Wonderful . So, let's begin. It's no surprise that agents are the hottest topic in AI right now, but your agents are only as good as the data layer beneath them. Today, approximately 80% of all corporate data is in an unstructured format. They are not arranged in neat rows or columns.

`00:54` Why unstructured documents are the next data frontier

They are stored in documents, PDFs, videos, PowerPoints, and other formats. And this number is only growing. An estimated 2. 5 trillion new documents are published and created annually. The documents contain corporate context in almost every industry. Medical forms in healthcare, insurance claims, financial reports, etc. And the pattern is the same in all industries. There are important insights hidden in unstructured documents , and once you can unlock them and transform them into clean structured data, you can do a lot with them.
You can power agents, create dashboards, chatbots, and more. But many companies still struggle to unlock their unstructured data. Over the past year, we've spoken to hundreds of clients from different industries, and we hear the same story over and over again . First, the documents are scattered across many different sources. SharePoint, Google Drive, email, SFTP and others. And our clients are trying to gather them in one place.

`01:42` Challenges of ingesting documents from many sources

When these documents are consolidated, it becomes even more difficult for them to get agents to work effectively with these often messy documents. And lastly: all of this usually breaks when scaled. The same problems you solve with 100 documents look completely different when you scale to a million. Ultimately, this makes it much more difficult to move quickly and launch new AI use cases. So today I'm just going to show you how to get value from your unstructured data more easily with Databricks.
For those of you who are not yet familiar, I want to introduce you to Lakeflow Connect. Lakeflow Connect is Databricks' own data ingestion platform. This platform makes it incredibly easy to migrate your external data in Databricks to your Lakehouse. Lakeflow Connect connectors are easy to use. Setting them up takes just a few steps, and they are fully manageable. This means that as long as we are able to resolve the issue, Databricks takes responsibility for fixing it so you don't have to worry when something goes wrong.

`02:31` What is Lakeflow Connect? Managed, incremental ingestion

This includes automatic schema change, data origin tracking, automatic retries, error handling, connector updates when third-party APIs change, SCD 2 type data change streams, and many other features aimed at reducing maintenance costs. Additionally, Lakehouse Connect is unified with the rest of your Lakehouse environment. This means you can perform ingestion in the same place you do orchestration, machine learning, and dashboarding, without learning a new platform, all under the full control of Unity Catalog.
And finally, our connectors are effective at all stages of work . Incremental reads, writes, and transformations ensure maximum efficiency of your pipelines when processing data. So, let's see how easy it really is to use LakeFlow Connect. Perfectly. Let's imagine that I am a data engineer. I was given the URL of a SharePoint folder full of invoice PDFs. Let's see what some of these PDFs look like. This is the classic type of invoice that we receive from a supplier.
You see tables, sometimes they are a little uneven, maybe it's a photo taken on a phone, different table formats, barcodes, numbers, logos, etc. If we look a few more times, we will see an invoice from another supplier. And maybe a third bill from a third supplier. Different vendors have different structures and formats, so it is very difficult to rigidly prescribe business logic for parsing these documents. This is a classic example that we often hear from our clients regarding document management.
So, let's see how easy it is to actually port this to Databricks. First of all, I need to go to the add data page. I scroll down until I find the SharePoint tile. I press on it. Now I'm taken to a five-step setup wizard. The first step is to select a UC connection. UC Connections are objects managed by Unity Catalog that contain credentials to access your external data sources. In this case, I can either choose from the UC connections already available to me or click "create a connection."

`05:01` Set up a SharePoint connector in five steps

Creating a new UC connection is very simple. All I need to do is give it a name, copy and paste the client ID for my SharePoint, and specify the URL domain for my SharePoint data source. Then through OAuth I just need to log in and I can save that UC connection so Databricks can use my credentials to access my data . Now, to save time, I have already filled out the setup wizard in advance. So, step one: I simply selected the UC connection that I had already created.
Step two: I simply give my data intake pipeline a name. I choose a destination for my event log. And what we see here is the ability to choose between structured tables or unstructured data. If I were trying to import Excel , CSV, or Parquet files, I would choose Structured Tables. This way, I can create Delta tables that synchronize with an Excel file or a folder of CSV files. For this case, my files are PDFs, and I want to use them for AI tasks.
So I'll click on " unstructured data " and then move on to the next step. Here I just need to copy and paste the URL I got earlier. So the download will take a minute. And we see that I can view my source dataset right in Databricks. I can look at it and see, "Yes, this looks like the right folder." So I'll select the entire folder, give the target table a name, and move on to step four. In the fourth step, I simply select the target directory and schema.
And finally, I can set a schedule for how often I want this to run, as well as set notifications. As for the schedule, I can do it once a day or set up something more advanced. There are many different options. If you're familiar with our task interface, it's identical. And that's it . After that, I can just click "save and run pipeline ". Here is an example of what this pipeline looks like. And we see that it creates a complete data intake pipeline that populates the target stream table that I just defined.
On this pipeline monitoring page, I can track successful launches and share them with others . I can also set up notifications. This conveyor is located in the same place as all my other production conveyors. So, just through the pipelines page, I can track all my ETL pipelines, as well as all my data acquisition pipelines at a glance. If you want to take a look at what the target table looks like, here's what we see.
So, for each PDF file retrieved from a folder, we end up with a single file directory table. For each file that matches a string, we get information like the original URL, extended metadata like any special business logic, last updated time , special tags or labels added by your business users that you can use to search for files. But we also have a column called "content". And if we expand it, we'll see that for each row, that cell contains the entire file, saved in binary format.

`07:52` What lands in Delta: file metadata and binary content

I'll show you how easy it is to access this binary format shortly, but that's pretty much it. In just a few simple steps, I can add all my PDF data to Databricks. But now that the unstructured files have made it into Databricks, how do I parse them? This is where the AI parse document feature comes into play . AI parse document is one of Databricks' flagship AI features. With AI parse document, you can easily convert your unstructured files into structured formats to quickly extract valuable data from them .

`08:41` AI Parse Document for intelligent document processing

Using multimodal underlying models, AI parse document allows you to analyze text, extract tables, and work with graphs, and only one line of SQL is needed to call it. All you need to do is simply call the function, pass in that "content" column I described earlier, and that's it . So let's see a demo of what it looks like. Document analysis is extremely simple. All I have to do is Ouch. Go to the agents tab. Then select "Create Agent".

`09:32` Demo: parsing PDFs with bounding boxes and confidence scores

Document analysis. And then I'll just select that raw file table that I created earlier. Here we see that it automatically selects the column that contains the contents of the file. And then I can click "Start Analysis." This will take some time. So I've already analyzed this in advance so you don't have to wait long. But here's what we see. We now see a visualization of both the original PDF file and the document alongside a display of our newly recognized content.
Each element has its own bounding box, so we can easily verify the information ourselves. And also, if I click on JSON, that will be the actual output of the function itself. We get even more information, such as the type of each element, whether it's text, shapes, or images. We also get its content, as well as a confidence score , which we can use to validate the results, and also to see which ones may need manual intervention for re-validation.
I can review several of these documents. And when everything looks good, all I have to do is click "use agent" and I can run it either in SQL or in a laptop. And that's it. Now, I'm sure this looks good in a demo, but what about on an industrial scale? This is where the beauty of Lake Flow combined with intelligent document processing really begins to shine. As I mentioned earlier, our AI capabilities power the analytics agent you just saw.

`10:51` Incremental processing of new and changed documents at scale

AI functions are research-optimized building blocks that can be easily deployed and chained together to apply these AI transformations at scale. They are automatically integrated with the intelligence of LakeFlow Spark declarative pipelines. This means that we can ensure that as new documents arrive, only new or changed documents are retrieved, analyzed, and then your subsequent pipelines are run to ensure the most efficient ETL from day one.
Yes. So , if I clicked on open SQL editor, now we see an example of a pipeline created using the AI analysis agent. To parse our documents, all we do is simply define a new stream table, call the AI parse document function, add that content column we got earlier, and define it as a new column called parsed. That's all. It probably took me a few seconds. Honestly, I could write this in one line if I wasn't too bothered by the formatting.
And what we see here is now the silver tablet that I have taken apart. We have the original table because I executed the select star statement, but now we have a new column called parsed that actually contains that JSON you saw earlier. Perfectly. And now that we have these structured tables, the whole world is open to us. You can use your parsed data like any other structured data in any table, dashboards, for traditional machine learning, agents, etc.

`12:28` Query parsed documents with Genie and dashboards

For this example, I will show you how to create a Genie agent. All I have to do is move here. Let me find, click on Genie. Click " Create". Select the table that was parsed earlier. Now I have my own Genie agent that contains only the contents of the dataset I chose . I can ask questions, for example: how much did we spend on hotels and flights? Now let's wait a bit. Genie will do its thing. He will think. It will try to perform queries and calculations to give you an accurate answer.
To save time, I already asked this question a few hours ago. So, we see that Genie was able to calculate the total costs from different providers to pay for flights and hotels. He identified this across three different accounts, provided the account numbers, totals, and if desired, we can examine Genie's entire thought process and all the queries he ran trying to interpret this parsed document data. That's all. Here's how easy it was to create your own Genie agent based on a single dataset from SharePoint.
As I mentioned earlier, this is all orchestrated through LakeFlow. This means that every time my data acquisition pipeline runs , it only checks: "Which files are new or which files have changed?" Add only these changes. Then the next pipeline will automatically start, which will process only new documents, and finally Genie will automatically refresh as its source data table is populated with new rows. I want to emphasize that Genie is just one of many ready-made solutions that can be easily deployed on Databricks.
Once your documents are sorted, limitless possibilities open up before you. Use the data however you want. Achieve any of your goals. Here are just a few options our customers are using today, but the data belongs to you. You yourself know how to use them best . So, to recap: today we showed you two extremely powerful tools that make it easy to load unstructured data and harness its value. First, I showed how LakeFlow Connect can easily set up download pipelines from external sources and bring them into your lakehouse.
And then, using our AI functions, in particular AI parse document, we showed how you can easily turn them into structured data for further use in any tasks. Regarding the future of LakeFlow Connect , we are adding hundreds more sources. In the coming weeks and months, we will have OneDrive, Box, and Dropbox, which you can see in the roadmap. We already have Google Drive, S3, SFTP, Outlook email, and many other sources where unstructured documents are often stored.

`15:33` Roadmap: OneDrive, Box, Dropbox and file-level permissions

We are also working on built-in permission handling. This means we will be able to inherit file-level permissions from your sources and automatically apply them to the Unity Catalog so your access control remains intact . Finally, we are also working on an intelligent user interface using the capabilities of Genie code. In the area of document analysis, we are developing built-in guided scoring, a better Genie code interface, an API for real-time processing, and soon— audio processing.
Good. I realize I've dumped a lot of information on you today . But don't worry. We've created an online demo that you can easily follow: copy and paste the same code, click where we are, and even pause me , which you can't do in real life. And also, I think there will be a more detailed analysis right after this session, although I understand that it may be difficult for you to get there right away. We have also posted all the materials from these in-depth analyses online so you can refer to them, share with the team, etc.
Perfectly. Thank you very much . Uh, thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.