From Car Testing to Lakehouse: How AVL Modernizes Measurement Data Analytics with Databricks

Summary

Automotive data explosion: This video explores how software-defined and autonomous vehicles generate massive measurement data volumes, requiring modern lakehouse infrastructure to handle testing, simulation, and engineering analysis.
Open-source lakehouse solution: Learn how AVL built an open-source lakehouse using Databricks and the Impulse framework to manage measurement data, enabling efficient engineering workflows for development and testing.
Data governance at scale: Discover how AVL implements data governance, quality assurance, and democratization strategies to make measurement data accessible to engineers while maintaining reliability and security.

From Car Testing to Lakehouse: How AVL Modernizes Measurement Data Analytics with Databricks

The car is becoming a device that updates like your phone, so the idea of "fully testing" one before it ships is quietly falling apart. How do you validate a vehicle that adapts every minute it's on the road?
In episode 2 of Data + AI Exchange, Viktoria Semaan sits down at AVL's headquarters in Graz with VP of Corporate Strategy Georg List and Director of Global Enterprise Architecture Andreas Juffinger to unpack how the automotive industry's shift to software-defined vehicles created a data explosion and how AVL and Databricks built an open-source lakehouse for measurement data to handle it.
⏱️ Inside the conversation:
🔗 Resources to get started:
Impulse repo: https://github.com/databrickslabs/impulse
Docs: https://databrickslabs.github.io/impulse/

Chapters

FAQs

Why do modern vehicles generate so much more data?

This video explains that software-defined vehicles fundamentally change development: instead of static hardware with fixed software, modern vehicles have software that adapts during operation. This requires massive data collection from test stands and simulations to validate vehicle performance, safety, and comfort across dynamic conditions.

What is the Impulse framework and how does AVL use it?

Impulse is an open-source framework built on Databricks for managing measurement data workflows. AVL uses it to handle the data explosion from development and testing, providing a structured approach to ingestion, transformation, and analysis of automotive measurement data at scale.

How does the lakehouse approach improve automotive engineering?

Traditional silos separate testing data, simulation data, and analytics. The lakehouse unifies these, enabling engineers to interpret data faster, iterate development cycles quicker, and make better decisions. This results in better engines, safer vehicles, and more comfortable driving experiences.

What's the future of data in automotive?

The video discusses emerging opportunities including autonomous agents that co-pilot AI systems, generative AI applications for vehicle development, and getting started with lakehouse approaches. These advances position Databricks as central to the automotive industry's data-driven transformation.

Full transcript

`0:00` The Paradigm Shift in the Automotive Industry

Automotive AI, they need a supervisor. The agent is the co-pilot. They can build it's engineers are problem solvers. >> I'm here at the headquarters of the company whose technology is being utilized for development and testing nearly every vehicle. Yet, most people don't know about them. Their name is AVL 2,300 it's so exciting to be here at the headquarters of AVL. I think what will surprise a lot of people, despite over road, most people never heard about AVL. would love to know what AVL actually does. >> We're quite okay with the general So, what we do is we're mostly in propulsion systems, so engines and batteries and uh gearboxes and e-motors and so on.

`1:01` What Does AVL Do?

And uh we develop them for our customers, we test them for our customers, we simulate them for our customers. And we sell our customers all the tools that you need for the development. So, in that um we working with most of our with most of the OEMs in the world. All the major ones in China, all the major ones in Europe, all the major ones in the US. Many startups and tech companies also that are new to the automotive field.
Uh and also uh the biggest suppliers, yeah, that for example create batteries for the OEMs. >> Automotive industry in general is going through massive transformation now with autonomous driving, software-defined vehicles. There's just so much more data being generated by those new systems. So, I'm wondering how it changed for AVL in terms of what you need to do now to test those new types of vehicles with new stack. >> Today, the software stack has become a lot bigger and software demands even more data for development.

`2:25` Navigating Industry Transformation and Data Challenges

Both test stand and virtual simulation are massive data producers. And for engineering then you have to interpret the data, you have to know what to do with it, improvements, iterations of the development. And uh and the outcome is a better engine and a safer vehicle and a more comfortable drive, etc. , etc. >> For someone who doesn't know what's AI-defined vehicle and software-defined vehicle, can you give a a one-sentence? >> You change developing a car from developing a hardware and adding the software to it to actually separating hardware from software and uh developing a software that enables a smart hardware to do many things.
Yeah? So, it's a very different way of thinking about the car as kind of a device like your iPhone. And if you disconnect the two, then then um you can actually add functionality or improve the functionality without changing the hardware. In the traditional vehicles, you have 100 computing units inside. And synchronize. They're very difficult to make them work together, etc. , etc. But if you have a software platform that on which you build functionality, then the platform is hugely scalable.
But we will have to change the way we do validation. Um the software stays with the car. The software, like with your iPhone, the software stays with the iPhone, and it gets an update every week. Well, if you go once a year to a garage and let your car be tested, but you have an update once a week, then this this completely meaningless, right? This once-a-year testing. >> If in the past I read a bit that, you know, you utilized the MATLAB, right?
But now you have just so much exponentially more data that's been generated, like petabytes of data. So, traditional um desktops can no longer handle it. So, I'm wondering how you navigated this challenge. >> If we look at the project today, just for it's not development, but just for testing and a little bit of simulation, we probably 1 to 2 terabyte by project. And that's already I will say, that's already in the past, yeah.
And only a small portion of the data is actually relevant. Within thousands of hours of data, there's probably only one or two or three instances that you really need. >> So, you have this division that's called AVL Racetech, and you work with Formula 1, Formula E, NASCAR. This is so exciting because you are literally engineering at the edge, right? Where milliseconds and millivolts matter. So, I'm wondering um what is it that you are innovating in this space, and how this feeds back to traditional cars. >> And the cool thing is in RaceTech, you don't have to prove uh so much the cost.

`5:25` Innovation in RaceTech and its Transfer to Traditional Cars

If you say, "Okay, I'm going to bring you a second on the on the track." You have a customer. And they know what it's worth to them uh to have a second to win a second on the on the track. And on one hand, um for example, what has come from RaceTech into normal engines is the efficiency of the engine has gone up by huge amounts. Yeah? Uh it was at 30-ish percent, yeah, in in the past. >> only use so much uh fuel, and you can only have so and so many engines per per year.
And so, these regulations put certain constraints that they had to increase efficiency so much uh and so quickly to win races. And that they've moved from 30% to nearly 50%, which is sort of limits of physics at a certain point. >> That's really fascinating, because we always think about regulation as something that slows down, right, innovation. But in this case, it actually pushed forward. >> They they think very hard about thing about regulations that yeah, make it difficult, and make it make innovation necessary, so that these teams, not only the driver, but also the cars have something to offer.
It racing uses a simulation that we call driver in the loop, uh where you can put a driver in a cockpit, and you move that cockpit around, so he really has the feeling he's sitting in the real cockpit, yeah? But he's driving a car on a screen, like in gaming, yeah? >> Virtual reality. >> Virtual reality, and driving that while his seat goes, you know, the real stuff, and the car is connected to it. And so, you can actually test the car, you can improve the car on the real track virtually with the real driver. >> Of course, now everybody's talking about AI.

`7:47` The Future Vision: Adaptive Cars and AI-Driven Testing

So, I'm wondering from your perspective 5 years from now, let's say a car rolls off the line, how much of this test will be driven by AI? >> What we're going to do is we're going to shape this 5 years. I think the whole the paradigm shift is There isn't this point in time rolling off the line or, you know, launching a car or periodic testing. There isn't a point in time that is important in the future because cars adapt.
Cars adapt every minute. Cars adapt as you're driving them because they see, okay, you know, if I make this change, it's more comfortable. Oops, there is safety issue out there I have to remember. So, they're going to adapt themselves all the time. I think we will need to think about how we do not neglect risk, doesn't exist. >> How would you suggest for other companies to get started with impulse and adopted? And perhaps also contribute. >> Use your time for value-added, you know, where where you have your knowledge and where you have your and what customers pay you for rather than maintaining things that you can buy and that are well maintained by your partners. >>, so exciting now to be with you after hearing more about the transformation that AVL is navigating in this exciting AI era.

`9:20` Technical Walkthrough: Building a Lakehouse for Measurement Data

I couldn't think about a better person to have this conversation. >> Thank you for having me. >> Gorg mentioned about impulse and how it was built really to be a lake house for measurement data to innovate how you do testing. I would like to understand more from architecture solution perspective how it was built and why Databricks was the choice you made. >> Stuff has been built You build a component for bricks, a build for an engine, you build the gearbox and all this kind of stuff.
And each of these domains had and has its own tools. So, naturally we have the problem of a lot of tools and data proprietary silo data in along that development process. So, we have a lot of diverse tools with a lot of diverse data formats, but coming from the other side that data without context is basically useless. Yeah. If you don't know why is that spike there? Was it raining? Was it something else? Yeah. So, you need to know where or how to put that data into context.
So, we're coming from the bottom from the data lake side and from the top with the data warehouse which is kind of in combination, tada, the lake house. Databricks is the most major player in that concept around lake houses and we've seen that being applied already with a lot of our as AVL, but also in my personal history I saw a lot of people using Databricks and it's a really good technical choice to actually start with it and work it.
So, it's the architectural natural solution to that problem. So, while there might be that one tool which is very great for that specific engineering task, yeah. A time series database and so on. It doesn't help you because it doesn't solve the problem. It solve does not solve the problem of the overall context and providing the engineer the complete information which is required to make the right decision. So that's what really Databricks and the Impulse framework because the Impulse framework the library actually allows us to handle now time series data in an efficient effective way on Databricks, yeah?
So this is this this was a kind of a little bit a missing gap and that's why we are working on that together with Databricks since basically years on on it so that to make that happen so that we can really have that platform where we can bring the data and the information together to solve the problems for our customers. That data is not created in the cloud. We need to bring it there. Impulse is that great framework opinionated with a data model and all the tooling what you need.

`12:19` The Power of the Impulse Framework

A very excellent trade-off between compute versus storage. It actually also adds that capabilities around governing it. AVL can really help to bring the data from the source where it is or where it actually originates into the lake house, yeah? So that Impulse can pick it up from here. Supporting 500 formats on the left hand side, bringing it to the formats which are out of the box supported from Impulse and then taking it from there and making it analytics ready. >> One thing when I read about Impulse that really stood out to me that basically in the past to for engineers to work on this test, it required a lot of this custom software like MATLAB, right?
But now they can run code. >> The data premises tools or with their laptops analytics, yeah? And if you have all that data in one place, yeah? Then you understand the engineering problem of maybe something like a battery management system or some specific engineering task, yeah? And you build certain capabilities into that scripts and so on. Now, if you bring them together, of course it's just >> search for that. We have centralized it into the single source of truth for I don't have one laptop, yeah?
I can use tons of cluster and machines and a lot of CPUs, yeah? >> I can give access to the right people at the right point in time, yeah? And of course, bringing the data from its raw format in a propriety thing into um the medallion architecture basically along these steps, yeah, you need an orchestration, yeah. Uh, a lot of things can go wrong, yeah, along the processing. A lot of unexpected data points could be in there and so on so that might kind of need to fix something and you need to do do that, you need to have transparency about quality all the time.

`14:52` Data Governance, Quality, and Democratization

So, you need to know where the data comes from. So, lineage is in there. And now for the framework, the impulse framework, it footprint is relatively small, yeah. So, you can access it with a little cost as as necessary, yeah. >> Okay, that's exciting. I'm wondering if you faced >> any challenges uh from maybe engineers that are just too used to work with traditional tools like MATLAB to adopt this new technology and new ways of working. >> If I learned one thing with AVL, then it's engineers are problem solvers.
And give them access to data on one place fully governed, you have the authorized authorized system behind that you know you can do that here and then you can play with the data, yeah. We thought of it so okay, initially you know with digital transformation and there is a you need a cultural change program, you need to push it into the people and tell them how great data is, yeah. In our case we said hey, wait a bit.
We are not yet ready. That data is not yet there. We need that one, yeah. So, uh slow and then give them just the possibility to access data and then say okay, and then they can figure it out for themselves. And this is is so great with this whole framework also. And I think it's really completely democratizing data. So, in that data needs to be findable. It needs to be accessible, interoperable, reusable, yeah? You never know what people do with your with the data they have access to, and maybe they solve the next big engineering problem uh just now over there. >> What's next for you?
Are you building any exciting agentic solutions? Are you utilizing Genie uh to write code for you, build applications? What are >> We can't hold them back. Now, people all uh all over the place just wait as a really waited for using uh Genie and so on so because explain me my data. And Genie has access to the data team Genie has access to the lineage. I strongly believe, yeah, generative AI, yeah, on agents, yeah, they need a supervisor, yeah.

`17:33` Future Outlook: Agents, Generative AI, and Getting Started

The agent is the co-pilot, yeah. Pretty much does the hard work, yeah? But, the pilot always can say, "No, that direction. We need to do it this way." Or that's not not good enough. If we have that data accessible to Genie in much less time. >> So, we talked a lot uh today about the future vision. We talked what Impulse is really enabling. Um in my opinion, the best way to learn is by doing. What uh would you advise for somebody who's watching this video?
How they can go ahead and get started? can do, yeah? There are sample data available in this kind of aspect to see you you that you can actually get a feeling for it, yeah? As an engineer, I would say get a feeling for it, yeah? Uh not just uh try to read the documentation pull the Git repository maybe fork it and think about on how you can actually work with it on a Databricks environment and then work with it and play looking for ways to get started, find the resources in

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.