Accelerating Model Development and Fine-Tuning on Databricks with TwelveLabs

Summary

• Multimodal model scaling: NVIDIA NeMo Framework and Megatron-LM on Databricks streamline workflows for pre-training and fine-tuning LLMs and multimodal architectures at scale with GPU acceleration and advanced parallelism.
• Context parallelism innovation: Leverage NeMo's context parallelism capability to parallelize over sequence lengths, making it ideal for video datasets with large embeddings and long context windows.
• TwelveLabs case study: TwelveLabs built the Pegasus-1 video foundation model using NeMo on Databricks, demonstrating how efficient data management and computational power enable scalable multimodal AI development from text to video processing.

Accelerating Model Development and Fine-Tuning on Databricks with TwelveLabs

Talk by Aiden Lee, Chief Technology Officer & Co-Founder, Twelve Labs, Inc ; Wenwen Gao, Product Manager, NVIDIA
Scaling large language models (LLMs) and multimodal architectures requires efficient data management and computational power. NVIDIA NeMo Framework Megatron-LM on Databricks is an open source solution that integrates GPU acceleration and advanced parallelism with Databricks Delta Lakehouse, streamlining workflows for pre-training and fine-tuning models at scale. this video highlights context parallelism, a unique NeMo capability for parallelizing over sequence lengths, making it ideal for video datasets with large embeddings. Through the case study of TwelveLabs’ Pegasus-1 model, learn how NeMo empowers scalable multimodal AI development, from text to video processing, setting a new standard for LLM workflows.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction to Twelve Labs
• 3:41 The video search challenge
• 6:13 Language interfaces for video
• 7:28 Core models: Marengo and Pegasus
• 8:40 Challenges in video training
• 11:30 Data layer and pipeline
• 15:21 Model architecture design
• 17:19 Performance and results
• 18:42 Implementing Video RAG
• 20:08 Real-world impact and conclusion

FAQs

What is NVIDIA NeMo Framework?

NeMo Framework is an open-source solution that integrates GPU acceleration and advanced parallelism techniques. When combined with Databricks Delta Lakehouse, it streamlines workflows for pre-training and fine-tuning large language models and multimodal architectures.

What is context parallelism?

Context parallelism is a unique NeMo capability that parallelizes computation over sequence lengths. This approach is especially valuable for video datasets where embeddings and context windows can be very large.

How does TwelveLabs use this architecture?

TwelveLabs built Pegasus-1, a video foundation model, using NeMo on Databricks. This case study demonstrates how the combination of efficient data management and computational power enables scalable multimodal AI development.

What are the benefits for LLM development?

Using NeMo on Databricks enables data scientists to focus on innovation rather than infrastructure. The platform combines efficient GPU utilization, advanced parallelism, and Delta Lake's reliability to accelerate model development.

Full transcript

0:00 Introduction to Twelve Labs

Thanks for Thanks for joining the session. This session is going to be jointly spoke by me and women from Nvidia. Um and I think we can get straight in. Um so quick intro of myself. Um. I'm one of the co-founders and CTO at Twelve Labs. And today I'm going to talk about video understanding um and a training recipe for one of our models uh Marengo, which is a native video video native embedding model, and go over that together with you.
So uh before we go into that uh wanted to give you a quick intro of our company of my of my of my startup. Uh we are Twelve Labs. We are a global full-stack video understanding research and product company. So we work with lots of videos. Uh we also do full-stack research and product, which means we do both both model training deployment and also product building available via APIs. Uh and you know we we're headquartered here in San Francisco with Seoul office and global offices around the world.
So we build uh we try to build world-leading video foundation models, uh which I'm going to get about into details. Um and um I'm very extremely privileged to work with amazing some of our amazing clients. Our clients are major sports leagues, world-leading content creators, uh the world's biggest security companies, and uh winning studios, and broadcasters. And um one of them I want to highlight today is MLSE, um Maple Leaf Sports & Entertainment.
For those of you who doesn't know uh MLSE, uh it's the owner of the Toronto Raptors and Toronto Maple Leafs uh and the teams. And they had they essentially have decades of raw footage of their players, uh games, and practice sessions, highlights. In their In their archive, uh we really want to create new contents for better fan engagement. And I'm going to talk about them more in details later in my slide. Uh we are also a Series A company.
107 We We have raised 107 million dollars in total. And PIA is one of our partner. We have been Here's to talk about how we work together closely. They help us build and train more efficient video language models. Uh We're also, you know, amazing partner Databricks. They power our GPUs and training infrastructure. Their training platform is extremely easy to use for So, for those of you who haven't tried tried it out, it's awesome.
So, try it out. Please try it out. And we have ThoughtSpot to combine with our enterprise SaaS software company. Uh you know, we are team of 60 plus scientists and engineers who marries video and language. Um Our team previously achieved uh many wins over the last 10 years in language and video understanding areas. One of the thing One of the wins that I'm proud of is ICCV Value Challenge. Uh it's in 2021, which was one of the first video retrieval video search competition hosted by Microsoft and ICCV.
Um and and we were able to win the first place here. Um One of the thing that I want to highlight because it's because we participated we used the perks from our participating in an accelerator, Fixstars accelerator, and we were able to against won against tech giants. Um again, I'm very proud to be part of this team, uh scientists and engineers. These exceptional team members make our team's jobs and our mission possible.

3:41 The video search challenge

So, enough about our company. Going deep a little bit into video understanding. So, why does video understanding matter? It's because we believe video is the language everyone's first language. Think about your first memory. Um you don't you're not picturing words. It's more like you're replaying a part of your life um almost watching a video, right? Um it's the best data format. Video is the best data format that can that accurately represent um our physical world.
Um this is why video has become the dominant mode of preserving knowledge today. And it's also called world data these days. Um so to improve with the improvements in technologies to film, stream, and store videos, um it's a matter of fact that the video would explode in every industry um taking up 80 to 90% of the world's entire data. And that's why um understanding video is going to be more and more important. Um so what are we trying to solve here?
Um let's say you're a company or let's say you have a archive um of your favorite team um all the videos in the world uh of your favorite team or players. What kind of questions you want to ask um to this archive of videos, right? If you're a huge fan of San Francisco 49ers, uh for example, for me, I would ask something like show me all the scenes where, the legend from 49ers, score a touchdown against Seahawks. Or something like something on dry one, something like tell me key moments that happened on particular day.
So um given this archive, this this kind of operations exactly what you want to do because then you can uh you will be able to use the retrieve contents in the in in in your business in the in the context of your business. But um how many of you have been actually frustrated with zero result when you type in similar similar queries or search queries in Google Drive or, you know, your gallery, right? You don't you don't really get results these days.
Um and it's it's wasting lots of valuable video assets because you're not able to find and understand them. And one of the core reason is because these video these um retrieval nowadays is rely on tags. And tags doesn't work because when you submit a query, when you submit an instruction that you want to find, it's not the tag the tag itself does not match to that instruction. So, and this manual tagging is usually labor intensive, they're inconsistent, and not very effective on by on retrieving the content that you want to find.
So, new but in going fast forward to 2020 2021, new technology paradigm has emerged. Language as interface. In 2021, everybody knows the first GPT-3 model model came out. It It gave It gave segway to interact with text data through language. CLIP model in 2021 gave a segway to interact with image data with language. For us, we want to see what can we create a language interface for videos. So, we trained and deployed two two video language models.

6:13 Language interfaces for video

We call it Meringue and Pegasus. Meringue is a video embedding model that takes that makes that turns video into embeddings, essentially enabling archival video search. Video language model takes in a video prompt video as a prompt and able to perform question and answering on top of that. And with these two models, we believe video understanding can can be solved solved solved together. A quick demo of what we do using this search you do using these two embedding models, you can submit semantic text queries.

7:28 Core models: Marengo and Pegasus

Something like receiver catches catches the ball for touchdown. It's able to retrieve it instantly. Um, Chaplin attempting to walk down an escalator in vain. Um, these are all uh queries that you're able to submit on our archive of video without having to think about what tags you have used uh to in the in the earlier when you upload these videos. You're also able to uh generate summaries. Um, so video-to-text operations.
Able to get highlights. Um, so getting both timestamp and um chapters. And simple description as well. And today I want I want to focus on the first model, our Meringue model, and um walk through how we at the high level how we train this model. Uh, before we go into the training part, I want to um I want to share insights on what makes video understanding really challenging in terms in in technological perspective. Right?
If you look at this image, um it can be both sunset or sunrise. You you don't have an idea if just by looking at this image, you're you're not able to assess that. But through videos, you're able to clearly see that because video in itself is a multiple frames where it has temporal components inside as a native uh na- nature of the video, right? So, temporal understanding is one of the core consider core parts of the video data.

8:40 Challenges in video training

There's also uh audio components. Uh, sorry, the audio is not being shared here. But, um depending on what kind of music is being submitted into the video, um the impression of that video can be very different. Um, one can feel it's a very sorrow video or sad video, but the other can be very exciting video if you put in a par- party-like music into the video. So, another consideration that video understanding has is audio understanding.
Third is to online this video data, this continuous video data with text data. So, with this video data that's very continuous, it's audio visual data that's very continuous, you have to align that to text data. For example, if you want to ask questions like where and when is this video taken? Um you need to train the model so that this video you have to close the gap between the modality, these two modalities. So, modality alignment is another challenge that the model faces.
So, these three challenges are very specific to models, but there are also engineering challenges that needs to be considered. This is a screenshot that I took from the Databricks Databricks website. The more most powerful node P5 instances each composed of eight H100 node H100 GPUs has roughly around 25 terabytes of local storage. And this will be enough for training very large language models because I'm just taking this from the Llama 4 model, which is trained on Llama 33 30 trillion tokens in FP8.
Roughly that's around 25 to 30 terabytes. Which can fit into this single node of local storage of 25 terabytes. For videos other hands, if you want to train your model on 50 million hours of video, that's roughly 5 terabytes of video files. 150X of the storage you need for compared to language model. So, another access apart from the model is that engineering engineering wise there's a huge data scale challenge that you need to solve.
So, uh with this with these four challenges in your in the mind in your mind. How should how do we train video native embedding models? We start from our intuition was to start from CLIP model, which is image language model, which many of you already know, right? CLIP is composed of uh two components. One is a text encoder that takes in text and translate to embedding. Image encoder that translates image to embedding.

11:30 Data layer and pipeline

It's trained to align these two modalities um by making positive relevant image text pairs closer to each other and negative irrelevant image text pairs further from each other. Um contrastive loss. Very trained on very large scale image text pairs. Um now in order to expand that this CLIP model to videos, um I'll go one by one from the left to right from data layer to model layer to the train layer um to see to share show you what needs to be done.
So first in the data layer, um to expand to videos, um instead of image to text pairs, we need we need an equivalent set of high-quality video text pairs um to train the model. And so what does high-quality video text pair mean though, right? We can take some we can take some some learnings from CLIP model. Um there's been various scaling law uh report being being shared on CLIP model. What makes a very good what what is the scaling law for uh CLIP model?
So scale definitely matters um for video text pairs as well. So we need lots of data. Composition of these video text pairs also matter a lot. Um if you're if you're not familiar with data comp challenge, it's a challenge um designed to just designed to uh uh assess the composition of the image text pairs. So, given a given a fixed budget of compute and data number of data, um the challenge is to let the challenge let the participants change how these image-text pairs are composed within that fixed budget.
And recent studies have shown it can um depending on how you compose it, uh the accuracy can vary up to 10 per 10 plus percent. So, composition of video text pairs will matter, too. Lastly, the quality of these text descriptions will matter a lot. Um for example, given the same image, it's one thing to explain it in one sentence very in a very high level and another to describe it in details almost in paragraph. Um so, the same will apply for uh for a high-quality video text pairs.
So, with these three factors or learnings from CLIP in mind, um what we did is just apply the same learnings in video text pairs, right? We first acquired lots of data. So, we started from huge video text corpus, whether it's publicly available or privately or privately licensed. So, that's the scale part. We filtered it um to 100 million video text pairs um through various techniques. Here, we are we are thinking about that composition of video text pairs, how we are going to compose that video that 100 million uh video text pairs.
And then we recaptioned it um basically rewriting the text pairs to uh models where uh or human annotators to augment the existing 100 million uh video text pairs. And our final data set composition is basically a compo is a combination of these three uh of video text pairs, image text pairs, and audio text pairs all together to train train uh truly multimodal video language model video embedding model. And in the in this process um um Nvidia uh Nvidia has a tool called Nemo Creator, which accelerates the video processing significantly.

15:21 Model architecture design

Um if you have if you haven't checked it out, I strongly recommend to try it out. Uh it it can it uses GPU to process these videos and uh and catch and perform captioning or filtering uh operations on top on top of it. So, this is something that we have closely worked with Nvidia team to uh when we constructed our data data pipeline. Uh next is a modeler. Uh text encoder we're still going to use text as the interface.
So, we we're not we don't have to modify that one, but we need to be do need to change the image encoder because CLIP's image encoder does not have um the temporal understanding and it does not consider the temporal understanding and audio understanding part, which are both inherent nature of video. So, design a new model architecture that can do that. Um composed of three parts. First, a vision encoder that can take in multiple frames uh and map temporal relationship between the frames instead of a single frame vision encoder.
An audio encoder that captures both non-speech and speech semantics. And fusing fusing model that fuses the two modalities to come up with a final multimodal embedding that captures both audio and visual uh representation of a video. So, we modify that uh CLIP's image encoder into video native encoder train on top of large-scale video text pairs. And it's being trained on the same train train train objective, which is to uh make positive or relevant video text pairs closer to each other and make irrelevant negative uh negative video text pairs farther to each other, contrast basically contrast contrasted laws.

17:19 Performance and results

Trained on top of few hundred 800s data works 800 plus Nvidia cluster GPU cluster on top of it to make it really to make it to make it make the training process really efficient. And our results not talk I'm not talking about the I'm not I'm not talking about the Pegasus layer right now, but Vermaringo if compared to Google's multi multi mode embedding Vertex AI or Open AI's clip, we have the state of the art results.
We also evaluate in various aspects, not just on single benchmark, visual search, motion search, small object search, image search. All these aspects together evaluate on 72 different evaluation data set together. And got state of the art results. And through this video embedding model now you can what you can do is to run these sort of semantic text queries on an archive of videos. So, something like Spider-Man flies through a street and catches a car with his web.
Or even over even submit an image query like finding a logo finding a moment where certain logos appear semantically without having to extract logo from separate expert models. I haven't touched upon the Pegasus the video language model too much, but Pegasus model model essentially is to turn is to turn your unstructured video data into text so that you can um you can general you can perform question answering on top of these videos.

18:42 Implementing Video RAG

And the reason why this Pegasus layer is also important is when you think about the original problem of having thousand hours or million hours of video database. And if the user wants to submit a user user user wants to submit a prompt something like show me all the scenes where from 49ers scores a touchdown against Seahawks. Then she want you want she would want an answer machine that can answer this, right? Something like in video A from October 20th in 1994, scores four four touchdowns.
Or in video B, two touchdowns were scored at this moment and this moment, right? And if you're familiar with rag, um retrieval can rag works well for text. This is essentially rag for videos, right? You're trying to do retrieval, first getting the first getting the top K candidates. Um and based on that top K candidates, um the video language model answer uh answer to that user prompt. Um so this is essentially what we want to build.
So two components, video embedding model, video language model that enables uh rag rag workflow for videos. And through this uh pipeline, um NLC, again, one of our one of our customer, they achieved 98% efficiency in highlight creation. Um they're able to quickly find and retrieve the right moments for them to create uh create contents from their archive. Um that took 16 hours to 9 minutes. So, conclusion, um videos right now it takes 80% of the data, but the utility is close to zero because you can't find them and you can't understand them.

20:08 Real-world impact and conclusion

Still on a very video understanding is a very still is a still unsolved science and engineering problem. Last Last thing, you know, we made good progress, but last last to be a lot of progress needs to be made. Um and we believe to all of you we believe strong video embedding and language model with this system are fundamental uh blocks to solve them. So, if you're interested in solving this together, uh we're actively hiring.
Uh so, feel free to uh send me a note. Thanks.

Learn more about the Databricks Data and AI platform.