We Killed ETL: Unstructured.io Foundation

Summary

Unstructured.io Foundation eliminates traditional ETL by enabling AI agents to access enterprise documents wherever they live — source files stay in place and extracted data lands in ephemeral Postgres provisioned via Neon, with pointers back to the originals — so all enterprise data outside Databricks is now workable from within it.
Unlike the existing Unstructured transform pipeline approach, which moves chunked and embedded JSON into Delta tables and vector stores, Foundation provides on-demand context extraction with a custom ontology layer for domain-specific field definitions, multi-pass agent retrieval through an asset store, and MCP integration for calling Foundation from Claude and other agents.
With 80 million downloads, support for 74 file types, and customers at enterprise scale, Unstructured.io has shifted from being a PDF-to-JSON ETL tool to providing an agent-native document access layer that removes the maintenance burden of custom connectors, chunking strategies, and embedding pipelines.

We Killed ETL: Unstructured.io Foundation

You’ve known Unstructured as the company to power your RAG applications; first we transformed your PDFs to JSON, then we built ETL pipelines for delivering chunked, embedded JSON to Delta Tables that enabled LLMs with low-latency RAG. Now we’re changing the model entirely. Imagine if agents could take context from your whole enterprise, without having to copy every document into Databricks? Foundation builds enterprise-wide context on demand. Source documents stay where they are. Extracted data lands in ephemeral Postgres provisioned via Neon, with pointers for the agent back to the originals. All your data outside the Databricks ecosystem is now workable from inside it. This talk is for every team still maintaining a pipeline.
Talk By: Christopher Maddock, Head of Product and Engineering, Unstructured ;

Chapters

FAQs

What is Unstructured.io Foundation and how is it different from the original Unstructured product?

The original Unstructured product transforms documents — PDFs, PowerPoints, images, audio files — into a canonical JSON schema with chunked and embedded content delivered into Delta tables and vector stores for retrieval-augmented generation pipelines. Foundation, described in this video as 'killing ETL,' takes a different approach: instead of copying documents into Databricks, agents access enterprise documents in place through Foundation, which extracts the relevant context on demand and stores results in ephemeral Postgres provisioned via Neon with pointers back to the original source files. This means documents never need to be copied into the Databricks ecosystem for agents to reason over them.

What is the 'rat's nest' problem that Unstructured.io solves?

The rat's nest, as described in this video, is the typical state of enterprise document ingestion infrastructure: custom connectors to S3 buckets, hand-rolled JSON transformation code, custom header and footer stripping, manually chosen chunking algorithms, embedding models from various providers, and downstream connectors to vector databases — all stitched together with bespoke code that requires continuous maintenance as new models, chunking strategies, and retrieval patterns emerge. Every new embedding model release, chunking strategy update, or retrieval paradigm shift means another round of maintenance across 50 or more custom components. Unstructured.io provides a unified transform layer that handles all of this, replacing the rat's nest with a single governed pipeline.

How does the custom ontology feature work in Foundation?

Foundation's custom ontology injection allows organizations to define domain-specific fields and relationships that Foundation should extract from documents, rather than relying only on the default canonical schema. This means a legal team can define 'termination clause' and 'governing law' as extraction targets for contract documents, and Foundation will extract and structure those fields specifically for agent consumption in that domain. This video explains this as the key mechanism for making Foundation accurate and useful in specialized industry contexts where generic document extraction would miss the semantically important fields.

How does Foundation integrate with agents and tools like Claude?

Foundation exposes an MCP (Model Context Protocol) integration so that agents built on Claude and other platforms can call Foundation directly as a tool during their reasoning process, requesting document context on demand rather than querying a pre-built index. This video explains that Foundation also supports multi-pass agent retrieval through an asset store, where agents can refine their retrieval across multiple passes as they narrow in on the specific information they need. The combination of MCP integration and multi-pass retrieval makes Foundation suitable for complex agentic workflows where the information needed is not known in advance.

Full transcript

[00:08] Good afternoon, everybody. Hear me? Okay. >> Awesome. Uh, can we start with you doing me a big favor? If you got a hat and you wouldn't mind, would you just put it on so I can do a selfie with it? We spent a large portion of our venture capital
[00:20] money on those hats and it's nice for the VC guys see that they got some return. So, if you just wouldn't mind popping your hat on your head for one second and then I'll do a big selfie with you all. That would be amazing. I know it's harder with the headphones, so
[00:32] just close enough would be great. Thank you, Benny. >> Got you. Awesome. >> So, thank you all for coming. My name is Chris. I lead product and engineering
[00:46] for Unstructured. >> Genuinely, my biggest surprise of today was looking in this app and seeing that this event was full. There was a sellout. There's a line out the door
[00:59] still. Ever since I've moved to America, I've never felt more at home that this many people care about ETL, PDF files, and JSON files. I finally feel like I'm amongst my people.
[01:12] And Structure is four years old. We got lots of customers, 80 million downloads, and we got some fancy investors and won a lot of awards. >> Bit. The company exists to solve this problem
[01:24] to get unstructured data to Genai. Classic iceberg slides, PDFs, PowerPoint files, images, audio files, they're all lost somewhere in an enterprise where the plumbing that brings them to bear
[01:36] for Genai systems. >> Why is the device? >> Generally, what I encounter when we walk into an organization is this. We call it homebrew or sometimes lovingly and our
[01:48] customers aren't there, the rat's nest. What it means is that we walk to a customer and they say, "Hey, I've built a connector to an S3 bucket and then I used Pyandanda or Python docx. I transformed my file into a JSON
[02:01] document. I wrote a load of reax files. I did some custom SC screening. I took out the headers and the footers and I put it all together. Then I plugged in line chain, used a chunking algorithms. So I got my embedding models from
[02:13] bedrock and then I put a bunch of custom code in and I wrote a downstream connector to get this into pine cone vector database or for those at the moment neon
[02:31] >> this quickly will become what I've got 50 engineers maintaining a new model just dropped a new chunking strategy happened a new embedding model came out there's a new form of aentic retrieval yes you absolutely can build what instru structured supplies. My question isn't
[02:44] can you build it, it's why would you? At best, you'll put your 10 engineers on this and preparing your unstructured data for geni use cases. And what you'll end up with is a system that may have priority with mine and is costing you 10
[02:57] times more and a 600,000 lines of code that you need to maintain. So what we do in structured is essentially take all that away. On the left hand side we have a bunch of
[03:09] connectors to S3 buckets, Salesforce, SharePoint, Snowflake. If you want to move all your data from Snowflake to data bricks, we are supportive of that mode of data move CFKA as your
[03:21] everything you can imagine. We operate on 74 different file types and we transform everything into the same canonical JSON schema. There's multiple modes of transform for
[03:33] doing that. We have dedicated Python algorithms that we wrote ourselves. We have fine-tuned OCR and object detection models and we'll call out to VLMs as well with custom tune prompt and agentic
[03:46] recurrence over the data to get the most accurate transform out of it. >> We also offer built-in chunking algorithms. We can enrich the data doing density recognition, table extraction,
[03:58] all those good things. Connect to embedding models. And our product is this chunked embedded Jason. The most exciting product in the world. Thank you all again for coming and hearing about chunks embedded Jason. I genuinely
[04:10] genuinely feel like you are my people and you're still here. Once I did one of those talks to eight people and four people left after 5 minutes hurt a lot that moment did. So thank you for staying. Prody chunks embedded JSON. So this is
[04:24] beautiful for a developer whether we took PDFs, powerpoints, word documents, images, audio, same JSON schema that you can always work with. Then we know how to push that into downstream destinations with again 50 or so
[04:37] connectors for object storage, vector storage, >> speed up, >> graph storage and SQL storage. The flattener knows how to deal with it. That data is then consumed upstream by aentic systems, rag, LM applications or
[04:51] analytics applications. I care if I shook them. Thanks. >> Now, this is lightning talk. I only got 21 minutes total and I spent a minute trying to get a photo of the hats. So, instead of doing the demo and risking
[05:03] the tech gods, I'm going to show you in three slides the product demo that usually takes me 45 minutes. First up, transform is our product. On the left, you'll see somebody's resume with a load of bounding boxes that we've
[05:15] drawn around it. On the right, you'll see our canonical JSON schema with metadata containing coordinates of the bounding boxes of the element it took, the extracted text, a JSON ID, up to 30
[05:28] different pieces of metadata, including what language did it come from, what server did it come off, who had access to it. All this metadata is logged along it. And we also do structured data extraction from it as well. We'll pull
[05:40] out key value pairs >> that fits in with whatever ontology that you've got that delivers this data in the format that you need it structured in JSON.
[05:52] >> The connectors look like this. We bind sources to destinations. You just choose one. You off to it. You have a connector as a source. Maybe that's an Amazon S3 bucket. You switch to destination. You say this is all going into datab bricks
[06:06] delta tables. You've got a destination and then you bind sources and destinations together with a workflow. So this is a DAG direct cyclic graph that you'd see of any ETL tool. But in
[06:18] the past when ETL tools have worked off rows, columns and cells, we're working off the long form of off the unstructured text documents, PDFs, PowerPoints, Word documents. Can't believe it was.
[06:30] >> So that is our core product today, ETL for Genai. But our clickbait title that our CEO made me put, we killed ETL, is not that, right? We didn't come to hear that. We came to hear about what we're doing about actually killing ETL and the
[06:44] next generation of our products. >> 80% of the use cases that I'm faced with with customers is still ETL. It's like, I have this 10,000 research papers. I
[06:56] need to suck them out of S3. I need to convert them to JSON. I need to chunk them. I need to embed them. I need to write them to vector storage. and I need to do some kind of custom key value pair extraction and obscure some PII. Okay,
[07:09] so we do that a lot. What I've started here and now is yeah, Chris, we did that, but now we need to do this whole enterprises worth of data and it's cost prohibitive to hold two copies of every
[07:21] document in our organization, even if one's in JSON and one's in the original PDF or document X format. Yes. >> And then I'm calling embedding model every time. and calling a VLM every time. Unless the use case pays off,
[07:35] unless we know that this Geni Build is going to save us $600,000 a year or make $5 million in extra revenue, people are getting less and less willing to pay for rag solutions to hold two copies of the
[07:48] data. Even though unstructured works to keep those copies in sync with unstructured pipelines products that I just demonstrated to you, when we take data from an S3 bucket or a SharePoint
[08:00] folder, SharePoint's my best friend. Nobody likes having the data stuck in SharePoint. Thank you to everybody. If anyone's from SharePoint, I love you. Thanks for making that thing. You're going to pay for my next yacht.
[08:13] we pull. So when we pull that data out of SharePoint and convert it into embeddings in a vector store, we keep that data in sync. We know that this file was modified and we'll update it. So we're keeping this sync of the data, but we're still maintaining a copy of
[08:26] the data that's been transformed with the VLM maybe and that's had a bunch of embeddings made to it at massive scale. It becomes cost prohibitive
[08:38] >> for some of these people as well in very large cases. have a customer right now whose name I can't imagine who need to find a secret recipe that they've lost inside their massive organization. They don't care how latent this job is. They just need to know where the secret
[08:50] recipe is. >> So we started looking and thinking actually we need a new paradigm of this entire >> and what if what if we just left the
[09:02] files where we where they were. So, we heard about zero copy a lot and I've seen initiatives like that. But what if we like truly didn't move the files to make them available to Genai and we avoid the chunking, the embedding, the expensive LLM, the high
[09:15] token consumption and delivering it into a vector store and holding a duplicate of it. >> So, instead of that, we just left it where it was. So, this I don't think
[09:28] anybody's seen this. She might be the actual first people in the world to see this is what we call foundation and it's our new product. So the the idea is simple. What we're
[09:40] not doing is picking up every file in an organization, transforming it to JSON, chunking it, embedding it, and holding a copy of it in a vector store. What we're doing instead is using all of that core
[09:52] technology, that transform and those pipelines to take a peek inside every file. So imagine like a web crawler that crawls over every file inside of your enterprise. It utilizes those connectors
[10:04] to talk to all of those source systems. It knows how to open 75 different file types and it just takes a peek inside. It pulls out named entities. It pulls out topics and it'll pull out paragraph
[10:16] level summary or document level summary or page level summary. And what it does is build this entire thing that we call the asset store. Y. So think of this like a document catalog of every
[10:28] document that lives inside your organization. But whereas the traditional document catalog only stored the file, the location, and some governance information, we're actually telling you what's inside that as well.
[10:41] We'll also let you define a custom ontology about what's extracted from that document type. So if it's a financial document, maybe you care about profit and loss and expenditure, all
[10:53] being extracted alongside those. or in the case of our first customer on this product, does it contain the secret recipe? Yes or no. So, we build these massive cataloges and
[11:05] then we expose those to an agent. That agent has a bunch of rule sets exposed through Foundation's MCP server that lets upstream agents or humans do, >> it's a very happy looking agent over
[11:17] here, call into the catalog and request the data they need. So in the case of my shall not be named people, does this contain the secret recipe? If it does contain the secret recipe,
[11:29] just grab those documents that you need and process them on demand. So we leave everything where it is. We look inside everything. We extract the key information that we need to out of it. So we get everything we need to build an
[11:41] ontology. And now the agent can have multiple passes that search in the asset store. It can say, "Hey, do you have a secret recipe column?" No. Okay. Do you talk about sugar as one of the key ingredients? Yes. There's 12,000
[11:55] documents in your organization that look like this. Okay. Do you have any sweeteners mentioned as well? Yes. Oh, now there's 2,000 documents. And we net down exactly what the agent needs. If what the agent needs can fit into a
[12:08] context window, we deliver those files as is. If it's bigger than that, it >> the asset store is all built at top Neon. So we'll spin up an ephemeral neon database, post that data, transformed
[12:21] and demand just what you need into that and return that to the agentic use case. So now we're massively reducing the cost from early estimates in the early data we've pulled on this. This is something like 96 times cheaper than doing
[12:34] traditional rag and we're holding better levels of accuracy as well. So, what Foundation is essentially doing
[12:46] to us is it links our transform product. We've got a horrible invoice. It's a PDF. It's been scanned. It's bad. We'll transform it with Foundation. If you can't find it before because you had no way of accessing all this plumbing in
[12:58] your enterprise, we've cataloged it and we've stored it in the asset store. the cost high cost of running embedding models and VLMs to transform all this data at scale. We've seen massive
[13:10] reduction in that and massive reduction in token consumption as well. So here this is one one of our metrics is 93% less token consumption when you're doing it the foundation way and by the foundation way I mean we're no longer
[13:24] copy the file transformation JSON chunk embed it vector storage we index all that data we extract key information and we point the agent to the data that it needs on demand.
[13:36] So I'm going to ask you to remember 93% less tokens being used. It's important if you want any free drinks, food, and football tomorrow. >> So, how it integrates into Claude is
[13:48] super simple or anything that can call an MCP server. You jump into Claude, you press add tool, then you select unstructured. It's going to ask you to oth to whatever sources you want. We'll connect to those sources, and then we'll
[14:01] start crawling that data. It's super hard to see, but you can see without foundation on the left, with foundation on the right. without Foundation Claude can't find the
[14:13] files in a Google Drive that it's searching and it's having trouble with the size of the file exceeding the megabyte count and the scale that it's allowed to render into the context window. But with Foundation, those
[14:26] problems go away because we can we understand what's in every file. We can retrieve on demand. We can get all of your data to cloud or codeex or whatever your agent source is.
[14:39] We're nearly done. I still can't believe nobody left. Thank you for this. So, three products from unstructured to care about. Transform. All that does is take unstructured data and turn them into a canonical JSON. The pipelines, the thing
[14:51] we talked about at first, >> I see >> transform, chunk, embed, deliver metadata and JSON into a vector store. And foundation, the new thing that we're announcing today. So, we use this when
[15:03] we need to serve agents on demand. when you are cost sensitive but latency insensitive. So when you don't want a page chunk and embed everything at enterprise scale we'll just peek inside and when latency doesn't matter as much
[15:16] as it does with the pipelines use case during rack. >> So why the 93% happened? Firstly on the right uh our marketing team are awesome.
[15:30] We attempted to book a meeting room, >> but we accidentally booked the entire press club at the Four Seasons Hotel and we only found out that we did it last week. So now we've got this entire place where you can come. We're going to put
[15:43] the World Cup on. We're going to give away free drinks and free food and we're going to give demos of foundation. And I'm hoping if half of you would come to that, that would be awesome. You can have a place to sit down to this entire place that's super fancy in the Four
[15:56] Seasons. Everything's going to be free and we'll talk about everything we've talked about today and give demos. So, please come. It's a two-minute walk from the Four Seasons to the Moscone. I don't think the marketing people will be loud out again if they if they don't get
[16:09] people turn up to that. And then last of all, the QR code on the left, this is for Foundation. The product that I mentioned today that we're moving on to next after pipelines. This is in early access from today. It's
[16:22] gated, but we're looking for partners to work with us and develop this as we go. We're very keen on getting design partners that can help us bring this product to the next level. What happens is you'd sign up for foundation. My
[16:34] products and engineering team would help build the features that you want and set up your use cases. They would do all that for free. All we would ask is a commitment of an hour a week to come and feedback on what we're building, say how
[16:46] it's helping your use case and what's going on and help us drive to building a product that everybody loves. >> Fine. That is me. Thank you for staying. I greatly appreciate you all. And please
[16:58] come to the press club at Four Seasons tomorrow and come hang out. We'll all be there. We'll have more hats. Thank you all.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.