Unlocking Video Data at Scale: VLM Batch Inference with Ray on Databricks

Summary

This video presents a five-stage pipeline for distributed video analytics on the Databricks Data and AI platform: ingest video into Unity Catalog Volumes, register the Vision Language Model in MLflow, and run GPU-parallel batch inference using Ray actor pools and VLLM with Qwen2.5-VL-32B.
Unlike traditional computer vision that returns noun detection labels, a Vision Language Model produces timestamped narrative descriptions of what is actually happening, enabling interpretation rather than mere detection for use cases like retail loss prevention and manufacturing compliance.
A live retail demo processes 49 surveillance videos, flags 30 as suspicious, and confirms 22 through structured Delta Lake output, demonstrating how VLM prose can be parsed into queryable rows using Databricks AI Functions.

Unlocking Video Data at Scale: VLM Batch Inference with Ray on Databricks

Organisations sit on vast video archives (surveillance, manufacturing, inspections), yet lack scalable methods to extract insights. this video presents a production-ready architecture for distributed video analytics using Vision Language Models (VLMs) on Databricks.
We'll walk through a three-stage accelerator: (1) video ingestion into Unity Catalog Volumes, (2) VLM registration with MLflow for reproducibility, and (3) distributed batch inference using Ray and VLLM with Qwen2.5-VL-32B. You'll see how Ray orchestrates GPU-accelerated inference across video datasets and how to extract structured entities from VLM outputs using Databricks AI Functions.
Attendees will leave with:
A reusable pattern for multi-modal video intelligence at scale
Working code integrating Ray, VLLM, and Unity Catalog
Prompt engineering techniques for video inputs
Cost and performance considerations for VLM workloads
Whether your use case is retail, manufacturing, or public safety—this pattern applies.
Talk By: Samantha Wise, Senior Specialist Solutions Engineer , Databricks ;

Chapters

FAQs

Why are enterprise video archives largely unanalyzed today?

Traditional computer vision tools return noun labels with confidence scores—detecting that a 'person' or 'bag' is present—but cannot interpret what is actually happening. This detection-without-understanding gap means most organizations cannot extract actionable insights from their video archives without expensive manual review.

How does the five-stage VLM pipeline work on Databricks?

The pipeline has five steps across three lanes: video files are ingested into Unity Catalog Volumes for governed storage; the VLM is registered in MLflow with a production alias for reproducibility; Ray actor pools distribute GPU-parallel frame processing; VLM prose output is parsed into structured Delta Lake rows using Databricks AI Functions; and results are served for downstream SQL queries.

What model is used for vision language inference in this architecture?

This video uses Qwen2.5-VL-32B served through VLLM on GPU compute within the Databricks Data and AI platform. The model generates timestamped natural-language descriptions for each video segment, which are then parsed into structured columns by a downstream Databricks AI Function for SQL-queryable output.

What does the retail loss prevention demo in this video show?

The demo processes 49 retail surveillance videos using the pipeline, flags 30 as containing suspicious behavior, and confirms 22 of those as actual incidents through the structured output. It demonstrates that interpretation-level analysis—identifying exactly what happened and when—is achievable on data that already exists in an organization's storage environment.

Full transcript

[00:08] Good afternoon everyone. I hope it's been uh a really exciting week and I know we're itching to see the chain smokers later. Um my name is Samantha Weise. I am a senior specialist solutions engineer. Um I'm based in the London office. My specialism is an an
[00:20] ML. The next 20 minutes we're going to be talking about uh the largest lease created asset that your business owns and what it takes to turn that into rows in a delta table.
[00:34] So just a disclaimer, anything that I describe as next is are people able to hear me? Yes. Okay. I think there's someone with a faulty
[00:46] headphone. Okay. Um but yeah, anything I describe as next is application architecture and not product roadmap.
[00:59] So picture a multi-sight retailer, thousands of locations, half a dozen cameras per store, and 25 gigabytes per camera per hour. If you run that math out to a year, you'll get several
[01:11] exabytes. Your data warehouse has answered a billion questions about your business. Your video archive has answered roughly zero.
[01:23] And the reason for that is that there's no primitive that turns what's happening in this footage into a row in a delta table. Today, we changed that using infrastructure you already pay for.
[01:37] So over the next few minutes, I'm going to give you a five-stage pattern that you can clone tomorrow, a live SQL query against a real result table, and five gotas with fixes so you don't have to
[01:49] discover them yourself. So this is what your computer vision pipeline tells you today. Person, bag, bottle, cashier, zone,
[02:03] nouns with confidence scores. These are useful for detection or useful for counting but useless for understanding. This is what a vision language model tells you
[02:16] sentences with timestamps. Person in light hoodie reaches behind counter at 831830. Conceals item in bag and 31910. Exits without scanning. Cashier did not
[02:29] appear to notice. It's the same video, same hardware budget. The gap between person and person concealed item than bypass checkout is the gap between detection and
[02:42] interpretation. For loss prevention, for safety, for compliance, interpretation is what you actually need. The trade-off is cost. And the rest of
[02:56] the talk is how we pay that cost. So how do you run that at scale on data that you can't move out of your environment? Five steps, three lanes.
[03:09] Top lane is storage and governance. Middle is compute and bottom is outputs. Try a skull to make that a little more visible. But that yeah,
[03:23] hopefully everyone can read. Um before going through this, four of these five steps are infrastructure that you already pay for. The new bit is the green box in the middle.
[03:37] So step one is Unity catalog volumes. So you put raw video bytes into a volume. It's the same governance posture as a delta table. So lineage ACL's audit
[03:51] CCTV is exactly where files in a bucket plus a metas store reference stops being good enough. Step two is a delta reference table. So
[04:03] one row per video you get path duration metadata using FF probe and UDF gives you duration and this is what our pipeline can filter on.
[04:16] And then step three is MLflow uh UD yeah MLflow and Unity catalog model registry. So the model we're looking at is Quen 2.5VL 32B
[04:28] registered with the production alias. So from here on the model is just one config string want to roll back AB test swap. It's one flag change.
[04:41] Middle lane we have rayon datab bricks multiGPU we have two stages a CPU stage which is wide and a GPU stage which is narrow
[04:54] same DAG different shapes ray expresses that natively spark udfs can't now looking at the bottom lane we have a
[05:06] delta table of free text VLM outputs makes it auditable joinable and step five AI functions. So it's one SQL call that turns that pros into
[05:18] queryable rows. That's the thing your dashboard wants. It's five steps and four already on your lakehouse.
[05:31] Two governance moves to start. We have video bytes that go into a UC volume. So lineage and ACL's come for free and the posics like path volumes cat schema v so stuff like ffmpeg ff
[05:45] probe anything that takes a p a path it just works there's no custom mount then the model goes into mlflow with production alias and we use the path
[05:58] form of log_model so we save the model and processor to a disk first and then log the path Reason for this is that the dict form just doesn't work for multimodel models.
[06:11] So the path form skips the validator. It's one line of difference and the registration completes. The net result is that the model is one config string. So we've got that quen_5_VL-32b
[06:28] production. A mouthful I know. But anyone running that inference resolves the alias. So roll back, roll forward. It's just one API change. So now the more interesting half is
[06:40] running 32 billion uh parameters across video at scale. So we have two map calls different resource specs.
[06:52] First is the CPU stage convert to prompt. So we have ffmpeg frame extraction and prompt assembly. We set concurrency from 10 to 24, one
[07:05] CPU per task. And we have wide concurrency that hides the ffmpeg latency. So if we look at the code, the second bit the filter, see we have a filter in
[07:17] between. So we drop rows where ffmpeg has failed because corrupt videos are real. So surface the failure and drop it before the GPU sees a none.
[07:29] And then we have the second map quen video processing two GPUs per actor tensor parallel equals two to shard the uh 32 billion parameter model and then concurrency 2
[07:42] to 12. Thing to remember here is the GPU is never the bottleneck. The pipe to the GPU is and ray is what straightens that pipe.
[07:56] Spark UDFs can't express two-stage hitter regenerous concurrency cleanly. Ray core can so it's two map calls and that's the whole architectural move.
[08:09] Another thing on this slide we have the repartition to 200 obviously tweaking according to the data set that you're working with. Some videos 30 seconds, others 20 minutes. Roughly four times the GPU
[08:22] concurrency. So wide enough to even out the tail latency on long videos, but narrow enough not to blow up the partition overhead.
[08:36] Now inside the GPU stage, VLLM is doing few things to make the 32 billion economic. And why you care is because these knobs are why this fits into two GPUs. instead of eight,
[08:48] you skip them. You're either paying for a bigger cluster or accepting a smaller model. So one tent of parallel size equals two. So the 32 billion model shards across
[09:02] two A100s. Without it, the model won't fit. Two GPU memory utilization at equals to 0.95. So we pack the GPU to 95%.
[09:16] Without page detention, and page attention is built into the LLM, you'd lose roughly a third of that to fragmentation. And then three, maximum sequences equals five. So we have five concurrent uh
[09:30] sequences per active sharing the cache. Continuous batching means that new requests join onto the next decode uh step and the padding waste goes to zero. So
[09:42] running this on 49 videos, two A100s, we had singledigit minutes of pure GPU time. Most of the wall clock is cluster boot and dependency install, not inference.
[10:00] So the VLM gives us pros. Our dashboard wants columns and there's exactly one SQL function between them. So we have AI query which is data bricks AI functions primitive. It calls the mod model serving endpoint
[10:14] from SQL. We pass it the pros the instruction and the response format a JSON schema. The schema is enforced in server side. So the endpoint cannot return
[10:26] non-conforming JSON. This is the difference between we usually get JSON and the column always parses. There's no triing set downstream. You might be thinking why have a
[10:39] separate LLM call instead of asking quen for the structured outputs. And the two reasons one is that the prose itself is a deliverable. So we keep it for human review and audit.
[10:52] And llama 3.3 70 billion from foundation model API is a cheaper structuring step than using Quen's expensive context on JSON syntax.
[11:09] So before I show you the rows, um demo over here, the data set that we're using is public on Kaggle. Um but the people didn't consent to being on a conference screen. The source is a YouTube channel. Actually, let me just make this bigger.
[11:24] Um the source is a YouTube uh channel that publishes um alleged shoplifting footage as content. Um so I'm not going to show any frames on the screen. Uh our repo attached does include a reduction notebook for production use but for this
[11:38] talk the structured rows are the demonstration. So we have this query over here that is looking at BLM outputs how it is structured. So if I run that hopefully it does load. Cool. So looking
[11:51] at the first row, we've got electronics and hardware store assessment confirmed with confidence one. Person one wearing a white shirt item from the shelf
[12:04] concealment under their arm. A time range 181529 to 18600. Uh let me see if I can get that back. Cool.
[12:16] So, of the 49 videos, we had 30 flagged and 22 confirmed at an average confidence score of 0.91. We had a 61% recall against the subset.
[12:29] Bit of a story behind those numbers. So, we started with binary incident confirmed true or false and we got 22 out of 49. And then we switched to a multiclass uh
[12:41] classification with uh confidence scores and we recovered four events that the binary had thrown away from uh suspected to confirmed uh which took us to 26. And then we cut a regression in our own
[12:54] cleanup. So we'd hardcoded 16 frames per video where the original recipe sampled adaptively. So uh one frame per second source. And so we reverted and then that meant that
[13:07] seven video seven events upgraded from likely to confirmed and then one more flagged overall which brought us to 27. And then uh with AI query there were three rows where uh it returned null
[13:19] even though the pros clearly described shoplifting behavior. So we added a retry pass with a simpler classification call and uh that recovered all three. So of the 49 videos we had 30 flagged and then of those 22 confirmed.
[13:41] So what does it cost and what breaks? Three things. So looking at throughput roughly two to four frames per second per A100 at 32 billion. 49 videos sampled adaptively at one frame per second per source. Two A100s.
[13:56] This is about 36 minutes of wall time. Single digit minutes of pure GPU time um because most of the wall is cluster boot and model staging two cost shape. So DBU per hour times
[14:11] wall clock. The cluster I used was about 22 DBUs per hour. Sledigit dollars per video at the size. And this would be less at production scale when you're advertising staging across many runs.
[14:25] So the lever is run length and not GPU count. So we're at bites five lessons. One, MLflow dick form fails for multimodal models. Um so in path form
[14:39] fixes it. Uh this was with the version of MLflow that we were testing with two VLM transitively upgrades ray past the runtime patches. So pin braay or replace
[14:53] from spark with from items. Three the schema of JSON of type name string return string. So use explicit strruct DDL.
[15:05] And then four uh the binary boolean schema can throw away suspected pros. So use a multiclass enum plus confidence. You put confirmed and suspected into alerts and then possible into a human
[15:18] review queue. And then five, the AI query strict schema occasionally returns null unclassifiable pros. So if you add a retry with a simple call, um we were
[15:31] able to recover 6% of recall on this run. The thing about this architecture is that almost none of it is video specific.
[15:46] The five the fivestep skeleton works uh for retail loss prevention uh safety incident detection on manufacturing line cameras, defect investigation in uh inspection footage, dash cap event tagging and fleets, content compliance
[15:59] and media archives and triage and medical imaging. The things that do change per use case is the prompt, the schema and the model architecture stays the same.
[16:13] And our repo also has parallel examples for image and audio. So same DAG, different model, the pattern is symmetric across modalities.
[16:27] So of the 19 videos that we did miss, we had roughly seven which are long videos with sparse sampling. So 16 frames evenly spaced across 15 minutes can miss a 5-second concealment. And then two are very short videos where
[16:39] the frame extractor collapsed into a single image. One was the wrong camera angle entirely. And then the other nine were um model conservatism. So those would go into a human review queue and
[16:51] not the alert channel. But this is what we get when we're working with the free public data set. Five things that we'd add next and what they'd get us. So motion triggered upsampling for long
[17:05] videos and we want to densify around motion peaks. Frame count floor for short videos. So never fewer than eight frames version prompts with MLflow prompt registry because re reproducibility
[17:18] shouldn't depend on someone not editing a notebook. And then confidence uh rooted human review. So below threshold goes into a queue and not the alert channel.
[17:30] and then an Evo harness with a golden set delta table. Without Eval, you can't make any quality claims and this is your highest leverage V2 investment. So you change a prompt and change the
[17:42] schema. The architecture stays. So your video archive is the largest data set that you uh don't query and the pattern to do that is in this repo. So
[17:54] if you clone it, change two things and you can run it on tomorrow's data. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.