Gaining Insight From Image Data in Databricks Using Multi-Modal Foundation Model API

Summary

• Multi-modal foundation models: Databricks provides a unified API to understand images using powerful foundation models within data workflows.
• Image insights at scale: Analyze, classify and extract insights from visual content like product images, documents and videos without specialized computer vision expertise.
• End-to-end pipelines: Build production workflows from image ingestion through model deployment, integrating image analysis with other data types for business insights.

Gaining Insight From Image Data in Databricks Using Multi-Modal Foundation Model API

Talk by Ankit Mathur, Engineering Lead, AI Serving, Databricks
Unlock the hidden potential in your image data without specialized computer vision expertise! this video explores how to leverage Databricks' multi-modal Foundation Model APIs to analyze, classify and extract insights from visual content. Learn how Databricks provides a unified API to understand images using powerful foundation models within your data workflows. Key takeaways: Implementing efficient workflows for image data processing within your Databricks lakehouse Understanding multi-modal foundation models for image understanding Integrating image analysis with other data types for business insights Using OpenAI-compatible APIs to query multi-modal models Building end-to-end pipelines from image ingestion to model deployment Whether analyzing product images, processing visual documents or building content moderation systems, you'll discover how to extract valuable insights from your image data within the Databricks ecosystem.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction to model serving
• 1:47 Beyond text-based LLM inference
• 4:01 Multimodal inference at scale
• 5:49 Document and PDF intelligence
• 6:49 Model architecture and APIs
• 8:42 Product demonstration
• 13:46 Future roadmap and conclusion

FAQs

What are multi-modal foundation models?

Multi-modal foundation models can understand and analyze multiple types of data, including images, text and structured information. Databricks provides OpenAI-compatible APIs to query these models, enabling organizations to extract insights from visual content within their data workflows.

What are common use cases for image analysis in Databricks?

Organizations can use image analysis for product image classification, processing visual documents like PDFs, building content moderation systems, and analyzing any visual data. This video demonstrates how to integrate image insights with structured data in the Databricks lakehouse for comprehensive business intelligence.

What is the best model serving platform?

This video presents Databricks Model Serving, which provides unified access to any model—classical, foundation models hosted by Databricks, or hosted elsewhere—through the same governance layer. The platform establishes common abstraction for querying different models with the same API, enabling seamless swaps between latest models. Multi-modal capabilities let organizations analyze images, text, and PDFs at scale without specialized expertise.

Do I need computer vision expertise to analyze images in Databricks?

No, this video shows how to leverage Databricks' multi-modal Foundation Model APIs without specialized computer vision expertise. The unified API abstracts the complexity of computer vision, allowing data teams to build image analysis workflows using familiar data processes.

How do you deploy image analysis models in production?

Databricks enables end-to-end pipelines from image ingestion to model deployment. Organizations can build production-ready image analysis workflows that integrate visual insights with other data types, creating comprehensive data products for business applications.

Full transcript

0:00 Introduction to model serving

Thanks for coming everyone. I'm the engineering lead for our model serving platform here at Data Bricks. Um, and I'm excited to chat with you all today about some new capabilities that we're introducing around multimodal inference in our foundation model API. So, let me first start by talking a little bit about what the product we've built does. Uh we've built this product called data bricks model serving. Our philosophy has always been that we want people to be able to access any kind of AI model uh whether it be a classical AI model, a foundation model that is hosted by data bricks or a foundation model hosted by anyone else uh from the same kind of product with the same uh single pane of glass and the same governance that you get within data bricks.
Uh so we want there to be one place for that and in particular for foundation models we wanted to be able to give people the opportunity to query all sorts of different kinds of models with the same API. Uh and this is pretty critical because the kinds of models with and the and the rate at which models are improving is so fast that if you're really building an amazing production genai application, you really need the ability to be able to swap to the latest and greatest without that much effort.
One way to do that is by establishing a common abstraction such that you can actually uh query those models, write your application code in one way and then be able to switch very seamlessly. So what we really want to do is be able to build the best integrated end-to-end serving platform. Uh and so I'm going to talk a little bit today about some new capabilities we're adding there. Uh so let's talk about what you can do with foundation models.
I think everyone has tried trackt is familiar with the ability to go from text in to text out. Uh and there's you know been obviously a lot of great progress in that area and a lot of improvements that people have made in texttoext inference. But this is not the whole story. most of the time, especially when interacting with real customers, uh the reality of what the data that's passed in is a lot more hairy than just text.

1:47 Beyond text-based LLM inference

So, this is kind of the easiest thing for an LLM to do. But when you look at the wide swath of use cases that things people can use for their foundation models, it includes a ton more stuff. For example, if you're in any industry where documents are either used now or were used before, PDFs are all over the place and frequently they're pretty hard to actually get in in real insights from because they're in a uh image.
They're in some format that's not just text. Uh contracts, RFTs, a lot of these kinds of things are like that. If you're in retail, you probably have a lot of data coming in with different kinds of images. For example, a lot of the retailers that we work with have product cataloges that have images. A lot more stuff is added every day, but the descriptions are never up to par and frequently aren't there. So, it's really hard for people to figure out from this like ocean of images, what can what's actually useful?
What are these images about? How do I semantically analyze these things? So in general what I've found uh especially within enterprises is that what Genai has unlocked is the ability to analyze and structure data at high scale where previously in order to do that you needed in order to do that with high intelligence you needed to scale with humans which obviously is extremely expensive and it meant that a lot of data within people's lake palaces was just untouched and unused and what foundation models have done is actually made that affordable. to analyze with high notes, there's PDFs, there's audio, there's video.
How do we actually make sense of these things with these same technologies? And so what we're announcing today uh available in data bricks workspaces this week is the ability to actually pass those images into LLM via the REST API. Uh this is going to work for a wide variety of models, but what it what it does is it unlocks the ability for people to build real-time applications that take in images and get outputs with text in very high quality.

4:01 Multimodal inference at scale

But like I said, right, like real-time applications are one set of places that people are able to use these foundation models. But how do I actually do this with all of this data that's sitting in the lakehouse? Do I need to like write some query that then you know sends a bunch of parallel network requests to a rest API? Uh I hope not. And so what we're actually also introducing at the same time is the ability to do this at massive scale.
Uh what we are what we the interface that we've chosen for this kind of a thing is a SQLbased interface that might be different if you're familiar with other batch products that exist in the market. They often ask you to upload a set of files somewhere. I mean, that kind of doesn't make sense for text, but it makes even less sense for images, uh, which are already in your lakehouse. So, with this interface, you're able to write SQL, write a quick prompt, point at a model, and go and run that at massive scale.
So, here, for example, we have a bunch of shoes. Let's say we wanted to filter our catalog for the ones that just are boots. I mean objectively if you had if you had to solve that problem from scratch you probably would have to go find some image classification model then deploy it then run stuff through it in order to answer this one question and if the question changed a little bit you'd have to go do a lot of that again with foundation models they offer the flexibility to do this once and change this over time very easily but like taking it a step even further right you have images but but actually PDFs are a whole different beast because you don't just want the text from them.

5:49 Document and PDF intelligence

They often have rich information like charts, graphs, uh, and without that information being processed, you actually don't have a good capability. Historically in the industry, there have been OCR tools. There have been a lot of different bespoke things. With foundation models, yet again, we have the opportunity to generalize and thus save costs and time. So, we're introducing this uh this uh new AI function called AI parse document.
And with that all you need to do is put your cat your uh PDFs in a catalog within data bricks and then use this thing. We actually extract markdown directly from the PDF using foundation models and our research team is currently using llama for multimodal but actually in the works we have a model that which is a custom model that does really really well on PDFs. So which models are we using right now? uh llama for maverick is the one that we use uh we've added support for in the open source but within data bricks our philosophy is we want our product to be the hub of where you come to build your genai applications and that means having the highest quality models available out there so what we're introducing what we introduced in March was the ability to have cloud models natively hosted in data bricks those are now multimodal capable tomorrow we'll announce the ability to actually use Gemini models in a first party within data bricks as well.

6:49 Model architecture and APIs

So we're really excited to bring all the highest intelligence models to data bricks and allow you to use them to process things uh to process images and PDFs and things like that. Just to take a quick step back what happens in a transformer when you actually do this kind of inference. So you may have heard the word token token like you know text can be converted into tokens by a tokenizer. What happens with images? Well, what's interesting is you obviously want the images to be in some kind of a standard shape.
So, we resize them and we pre-process them and then we actually take an encoder which is able to semantically map those images into token space uh by using embeddings. And so then what you end up having is those images get converted into tokens. We merge them together and we run the model as one. What's convenient about this approach is that there's not a second other model out there. you just have a single model that you need to train with high intelligence.
And so that's been really really impactful because it allows you to centralize your workflows across a single model. So we have this open AAI compatible API that's you know obviously great if you want to build a real-time application. All you need to do is pass in those images in the same format that you would use to query OpenAI and now you can use any model that's out there. So I want to actually uh take a bit uh and show you what this looks like within the product.

8:42 Product demonstration

So let's step out here over uh and uh we'll start with real time. So if you can see here within data bricks we have this playground where you're able to actually use a bunch of different models. You can use llama 4 clawed etla models. And so here we have llama form maverick selected. And all we did here was just add this ability to attach a file. So, let's say I upload this one. Uh, tell me about the shoes and how they're made and designed.
So, as you can see, Llama for Maverick's super fast model because it uses this mixture of expert uh mixture of experts architecture is able to really really quickly give you a detailed analysis of what kind of shoe this is. Uh, you can do this with any kind of really really highly complex image as well. Uh obviously your mileage may vary on quality, but we've found that these models are actually quite good uh at the vast majority of very simple use cases.
So how do I now take this thing and take it to the next level? How do I do this on a lot of data in my lakehouse? Well, all you need to do is write this kind of SQL. It's very simple. All you need to do is specify this model, specify your prompt, and in this case, what we're going to do is we actually give you the ability to do this thing where you extract a structured output. You extract a JSON output. This can be really valuable if you're building an application because your front end or your app can process that JSON in a in a way that actually allows you to render this in some kind of intelligent manner.
Uh and so what we do behind the scenes when we have this feature is we actually constrain the way that the model outputs its probabilities to fit the space that you're specifying in this schema over here. So we have a very simple schema over here. We're just asking like hey is this a boot or not? Uh and we've got about uh 200 images that we're analyzing here. Uh and so just for the sake of uh you know kind of moving fast uh we can actually just skip ahead over here.
And so here you can actually see that we were able to get not just the images which are stored in your lakehouse but also a very simple analysis on the on the right side which is a valid JSON and you can see that hey this is clearly uh it it's a boot or it's not and it's relatively accurate if you actually take a look at it. uh we were able to process all of that data in just 50 seconds. So that's a really exciting thing.
Uh one of the kind of value propositions of this SQL batch product is that it's able to deliver high throughput which means faster job completion times that allow you to integrate this kind of thing into your production ETL pipelines which already have SLAs's if they're business critical. So if we go back here we can see that we got the same result. Now, here's a cool thing that these foundation models are able to lock able to unlock for you this boolean value of whether it's a boot or not.
I mean, that's pretty simple, right? Like you could take a look at it. It's pretty easy to for you to do. It might not even be that much work. You could probably train a small classifier to do it. But if we just extend that problem a little bit, it becomes a lot harder. What if we make this a multiclass classification problem where actually we just want you to tell us what kind of shoe it is. Give me the categories. uh all you need to do is add that to this boolean here, run it again, and now the model is at work on a much more complex task.
Um now, if actually specifying the model or specifying the prompt is not something you want to do, another capability that we do offer is the abil is is a bunch of AI functions which are not uh model specific. You can do AI classify, AI analyze sentiment, all sorts of different functions that your SQL analysts can use without having to learn about the details of specific models or without having to do prompt engineering.
If these were PDFs, you could do spark uh if these were PDFs, you could use AI parse document and rather than converting it into, you know, a specific prompt with a specific text to output, we would actually already take the next step and use markdown. So the way to think about batch products in my opinion is try to try to find places that have a layered architecture where the lowest layer here is AI query. You can actually specify model and prompt.
The layer above that are AI functions which you know allow you to perform specific tasks and there you can actually slot in things like parsing documents that are actually a general purpose task. So here you can see the model the job completed and you can actually see okay cool like it's able to see these flip-flops and say okay these are flip-flops there's sandals uh there's cowboy boots there's regular boots there's slides you can get a lot more detail just by kind of changing a little bit what you're asking the model to do so that was a quick demo uh about how you can use data bricks uh for this kind of a task uh and we're really excited to make that available to everyone uh who has access to foundation model API this week.
So what's next uh in multimodal inference? Uh we want to go after giving you the ability to analyze all the data that's in their lakehouse. The next biggest thing that we've heard particularly from folks who interact with real customers is that they've got a lot of audio data in particular call transcripts that they want to generate insights from. Right now the process to convert a call transcript to text takes time and then you take the text and then you analyze that.

13:46 Future roadmap and conclusion

What if you could just directly analyze the audio uh without having that intermediate step? Those kinds of things are able to buy people leverage to do a lot more different kinds of things. Uh we also want you to be able to customize your model based on the kinds of data that you might see. These models are pretty good at the vast majority of things that you might see in the world, but very likely certain use cases in particular in manufacturing might need to uh might have pieces that you know honestly the model has never seen and so it's not going to be good at identifying those.
But we've actually found that with our agent brakes product and fine-tuning, you can actually just with some lightweight fine-tuning customize these models to be really good at specific tasks within your enterprise. Finally, uh we want to be able to make this a little bit more ergonomic for real-time customers. We understand that not everyone wants to pass in images as part of the request. So, we want to be able to allow you to specify those things as URLs uh that we get that we can then process and download on our side.
And finally, for any real customerf facing application that's using images, it's pretty critical to think about guardrails. Uh what kinds of images you're, you know, okay with processing and uh we want to add that directly to our AI gateway. So that's a little bit about how we are thinking about multimodal here at data bicks. Uh would love to, you know, kind of take a little bit of time. I I know there's actually we can do questions afterwards and they're pretty tight on schedule.
So, that's all I had and I'm going to be right here to answer any questions that people might have. Thank you so much.

Learn more about the Databricks Data and AI platform.