Building “Vehicle 360” at Daimler Truck & Bus — How We Find Needles in a 30,000 Parameter Haystack

Summary

• Daimler Truck and Bus built Vehicle 360 on the Databricks Data and AI platform to create a single, trusted, real-time view of more than one million connected vehicles across eight global brands, each transmitting more than 30,000 parameters and generating over 23 billion messages per month.
• Lakeflow Spark Declarative Pipelines handle incremental ingestion with a custom micro-batch merge strategy that processes only changed attributes per vehicle, making it practical to maintain an up-to-date view of ultra-wide schemas at production scale.
• Unity Catalog provides unified governance and easy data sharing across teams, while challenges around schema inference and LLM metadata handling on extremely wide schemas required careful struct object design and table architecture decisions.

Building “Vehicle 360” at Daimler Truck & Bus — How We Find Needles in a 30,000 Parameter Haystack

Daimler Truck & Bus built “Vehicle 360” to break down organizational silos on a global scale and create one trusted, up‑to‑date view of each vehicle. We combine configuration and production records, sales and aftersales processes, over‑the‑air updates and more than 300 GB of daily vehicle log files to maintain this view across more than 1,000,000 vehicles — each with 30,000+ parameters. On an open lakehouse architecture, Lakeflow Spark Declarative Pipelines keep data fresh and organized, while Unity Catalog provides unified governance and easy sharing. It gives teams one place to see a vehicle’s current state, drill into recent parameter changes, find vehicles with specific characteristics, orchestrate dynamic over‑the‑air updates, and diagnose issues faster. Beyond day‑one wins, this foundation also enables proactive anomaly detection, AI‑driven support and our journey to a software‑defined vehicle.
Talk By: Marco Sturm, Data & AI Consultant, Daimler Truck AG ;

Chapters

• 00:00 Introduction: Vehicle 360 and the 30,000-parameter problem
• 00:49 Daimler Truck & Bus: eight global brands
• 01:03 Global vehicle fleet: Freightliner, Mercedes-Benz, Thomas Built
• 04:29 Multi-brand data consolidation: separate source systems
• 08:41 Final table design: key attributes as columns
• 12:54 Schema inference and struct object handling
• 17:08 LLM metadata challenges on ultra-wide schemas
• 19:02 Lakeflow Spark Declarative Pipelines for incremental ingestion
• 19:15 Custom micro-batch merge: only changed attributes per vehicle

FAQs

What is Vehicle 360 and why did Daimler Truck and Bus build it?

Vehicle 360 is a single source of truth for vehicle data across Daimler Truck and Bus's more than one million connected vehicles spanning eight global brands, including Freightliner, Mercedes-Benz Trucks, and Thomas Built Buses. It was built to give product engineers, support teams, and operations teams one unified view of each vehicle's current configuration and state, replacing a fragmented brand-by-brand data landscape.

Why is the 30,000-parameter schema a significant engineering challenge?

Each connected vehicle emits more than 30,000 parameters, and the fleet generates over 23 billion messages per month. Managing schema inference and struct object handling at this scale required careful table design, and the extremely wide schema also created challenges for LLM-based metadata tools that struggle to reason across thousands of columns effectively.

How do Lakeflow Spark Declarative Pipelines support this use case?

Lakeflow Spark Declarative Pipelines handle incremental ingestion of vehicle data, applying a custom micro-batch merge that writes only the attributes that have changed since the last update. This approach keeps data fresh without reprocessing entire vehicle records on every pipeline run, which is essential when processing billions of daily messages across over a million vehicles.

What use cases does Vehicle 360 enable beyond basic data storage?

Vehicle 360 enables engineers and support staff to see a vehicle's current configuration, drill into recent parameter changes, and find vehicles with specific characteristics across the global fleet. The same foundation also supports dynamic over-the-air update orchestration, faster diagnostics for support cases, proactive anomaly detection, and AI-driven customer support capabilities.

Full transcript

[00:10] So, welcome everybody to my talk on building vehicle 360 at Daimler Truck and Bus. My name is Marcus Sturm and today I'll show you how we build a single source of truth for our vehicle data that enables us to find needles in a 30,000 parameter
[00:23] haystack. First, I'll give a short introduction to Daimler Truck and Bus before coming to the use cases we enabled through that and then showing the technical implementation and summarizing a bit
[00:36] with the learnings we are having. So, Daimler Truck and Bus. We do our job for all who keep the world moving. While you might not have heard of our
[00:49] company before, you have certainly saw seen our brands on the street. So, on your right, you have the school buses, the yellow school buses, Thomas Built Buses. That's one of our brands. In India, we
[01:03] have Bharat Benz. In In Europe, South America, we have Mercedes-Benz both for trucks and for buses. In the US, we have Freightliner, one of one the market leader on the highway and
[01:16] also Optimus Prime for those of you who have watched the Transformer movies. Then with Setra, we have another bus brand in Europe. And Freightliner is another vocational
[01:28] trucks here in the US. The last one on your left is Daimler Truck Financial Services who helps with financing and leasing our trucks. And in this company with many brands,
[01:43] I'm in the intelligent vehicle cloud platform department. We are the backbone for all of our digital services and have over 1 million connected trucks. And by having these connected and
[01:56] enabling all the different digital services you see in the bottom, we are processing over 23 billion messages a month. That's over 7,000 messages per second.
[02:08] And I have And to do that, we have built really high-level solution. The solution itself is not the focus for today, but the data analytics part for it. But to understand the challenges we
[02:21] have, I want to give a short introduction on where did we come from. What you see here, sample for trucks North America, we're collecting all these messages and then forwarding in
[02:33] the to the digital services. And at the same time showing it in an internal portal, which is needed for our product engineers, support engineers to understand build new products, configure
[02:45] the trucks, and then also if there's a support case, understand what's currently configured on the vehicle itself. In the past, that was what we have and then we started to use um Databricks as
[02:57] well. If you're interested in more than one vehicle, you want some analytics. That's why we have pushed all the service data from the multiple services we are running into Databricks as well and have the
[03:09] data. So then, our internal people could start analytics, but then again, okay, we are missing a lot of outside data. We don't know them yet, we have them maybe in some services, such as the sales data,
[03:23] which services are booked. When was the vehicle produced? What is the configuration of the vehicle? For example, is there a certain engine in the vehicle? All of this context information that's important to
[03:36] understand how our vehicle is configured and how it should behave right now. And the challenge now comes with if we are doing the analytics in our department,
[03:49] we not only have to understand our own data, but then again have to understand all the different other data sources that are missing the region. And quite often we have the same data point coming from multiple sources. So again,
[04:02] everybody has to learn um what is the best source for a data point, which one should I use, and can I trust the data from this source? And you have seen we have multiple
[04:15] brands, and we have grown through merge and acquisition. And that's why we have the same data points in all our different regions for all business units. But all of these regions, all of the source systems, are different. We bought
[04:29] the additional brand, we did not merge the source system together. And so to do analysis on the product on a global scale, we really have to understand not only one data source, one
[04:44] region, but all of them. And that's not really possible, and that's why we have introduced the vehicle 360. Uh single source of truth, the abstraction layer in between that really
[04:56] hides all of this complexity from the users in our department and the analytics people. And the use cases we are enabling with that, so high level some samples here.
[05:10] One is really the portal where we are now not only showing the current values of the truck in our systems, but we are extending it with all the different other details. When was the vehicle built? Which
[05:22] services are booked for the customers? Um what are the configurations? Do we have a certain engine in it? Then when it comes from this individual vehicle to I want to understand a fleet,
[05:35] I want to know what are the software versions of all the different vehicles in there. Or if it gets a bit more complicated to kind of identify what are the typical parameter settings my electronic control
[05:47] unit has on on the fleet is it rather used in this direction or in another direction? That's where we are using data bricks with secret code, but obviously in in
[05:59] the recent months and more and more also with AI-assisted analytics building there on top of that. Until now, the use case is always only about what is the current value of a
[06:11] vehicle. But then, we also wanted to have the history. Sample we had in the past that a customer complained suddenly the driver grading, which we are offering as service to them,
[06:23] changed. There was no change in the routes or the driver is doing the same, but it suddenly changed. And when we then looked into it, we could see, "Okay, few days back when you were reporting the issue, we had also then
[06:35] there a change in the ECU. So, our customers have their own workshops, especially here in the US, and their target is to get their vehicle on the road on the streets again as fast as
[06:47] possible. So, what they did, an issue in an ECU, they just took another spare one um from the workshop, put it in, did not remember to update the parameters so it fits to the vehicle. And that's why then the driver grading changed. And with
[07:00] this feasibility, we were able to see the changes we had in there. And not only can we see them for one vehicle, but straight away we can also look into why did it change or did you change it
[07:12] also on other vehicles? Do we expect the same issue on other vehicles as well? And what are now the tables we have the analytics tables in in the Unity catalog
[07:25] to enable these use cases? One is really this final analytics view. It's one table that is quite that is wide, where we have per vehicle one row.
[07:38] And we have simple values like the production date or the set of configurations, which cabin, which engine is in the S-list, so that would be the configuration. But then we also have more complex
[07:52] um elements, structured object, in these columns. For example, for the service, we store the list of services we have in there with some parameters in it. Is it active? Is it activating right now
[08:04] because the vehicle is not yet connected or we're waiting for the vehicle to connect again? And then we also to combine all the services in one column there. And if we are talking about electronic control units, they have hundreds of
[08:17] different parameters or even thousands of parameters. And also there we decided to put all the different information per ECU into one column. So, we don't have
[08:29] the challenge of going into 30,000 columns because then the table is not usable anymore. And whenever there's a new parameter, the metadata changes would would kill us and we can't maintain it, but we rather then
[08:41] focus on the important attributes, which we are putting as column in all of that. So, that's I would say the final table. The second table is really this change
[08:54] history. And there we have an entity attribute entity attribute value type of style. So, we have our entities, our vehicles, then we have the attributes, and we have
[09:07] the values for these attributes. Besides that, we have some more information that is relevant for for our maintaining, for our audit. One is which source did the value come from?
[09:19] Cuz as I have mentioned, sometimes we have multiple sources where the data is coming from. And we are really like we decided which is the best source per value. We are taking the values from there. If
[09:32] there's no value in here, we go to a second data source that has less high quality as a backup. And that's why we need the source in here. And then we have the start and the end
[09:44] for each row to know when where the values are valid. What you can see also in the first two rows for the service, we are putting not kind of the whole list of active services
[09:57] into one row because the service bookings, for example, change really per service on their own. And that's why we have this hierarchy in there. As the source system only gives us changes on
[10:09] individual service level, we then also have the changes to be on individual level and combine aggregated only then for the final table.
[10:22] So, how are we building these two tables? One, um how do we get to the change history? And here I've you see talked about it. We have many different sources where the
[10:34] data is coming from. There are streaming sources, but also batch sources. And streaming sources, it's rather easy. We have a change feed, so we can read all the changes from there directly.
[10:47] Batch sources, yes, so there's a daily update. Yes, if they would be doing a good job, we got a change feed what's changing on a daily base there. They did for simplicity of their data
[10:59] product most of the time a full refresh on each day. So, we get the full table each day. But then, luckily for us, we uh we can use a auto CDC from snapshot feature
[11:12] from from Databricks that's in their Spark declarative pipelines. That takes away and calculates now then the changes on our side. Once we have all the changes, we are transforming the relevant attributes
[11:25] directly into these entity attribute value rows. We have a temporary table there where we push all the changes. And from this temporary changes, we then really look into
[11:39] continuously, okay, we got a new change. Do we have this attribute already in the change history? If yes, did it come from the same source or from a different source of lower quality? If
[11:51] the previous source was lower quality, then we exchange it, swap to the new value. One of obviously, if it's a new value as well. But if you already have a high-quality value in there, but then a
[12:04] low priority source is updating the value, we ignore that change. So, we really only consider our most up-to-date or most the highest quality data source for all of that.
[12:18] Then, we're taking all of this data, we put the changes into the history, set the end date for the last value, set the new one with um start date and end
[12:30] date as none. And then, we have a change history of all our vehicles in there. This change history, we are then then taking
[12:42] and we are really looking, what are the changes? We take our target schema from the metadata. So, we know, hey, this um for
[12:54] example, a certain attribute, the services, that should be a list of struct objects. So, as we know that, we know, okay, for this, if we have an update in the service, we do have to look into the
[13:06] service history and get really like all the services right now, so we can go do the aggregation and calculate the new value. If it's something like only the production date, it's easy, we don't have to take any additional values into account.
[13:19] And then, we do for all the changes, so we do micro batching in here, calculate the pivot table, the table we need. And then we merge the certain columns
[13:32] into this our final table. And then we have our vehicle 360 table out of that in the end.
[13:45] So with that high high-level overview of the whole chain we have been working through or we have built. We have all the different data sources, streaming and batch ones. And we have really decoupled each source
[13:59] from um from the others towards the entity attribute value rows. That enables us one if one source has issues, we can just stop this individual task for for the time being.
[14:12] And I have talked that we have many different sources across the regions. We can really build this input source based on the region. So the sales system in the US we have a
[14:24] different connector or a different feed into the entity EFO into the first table compared to the sales system here in Europe. Then we have this decoupling done and
[14:36] from the second step onwards to the end, it's always the same again and across all the different regions we have. So then we have the consolidation taking
[14:49] changes from the highest priority source updating our change history. Again here it's really a long table so we can also look back that we have the same format in here. We can easily query
[15:01] what changed on a certain time frame, what changed in the last 7 days. Then we do the pivoting, the merging. Um which will then feed our wide table.
[15:13] What is the current status of the vehicle? Feeding our portal. But then also enabling all of the interesting questions for our analysts, for our engineering people
[15:25] to understand what is the current status in the field, what are the vehicles, how many services do we have active, and all of these questions.
[15:38] So, from building all of this and running that, what did we learn? And I think the first learning is not a surprising one, but nevertheless, I think the most important one. It was essential for us to understand the processes, how our data is running,
[15:52] to identify what is the good data source for all the different data points, and how how it fits and what's there coming from. And we have documented that in one place, so it's not only hidden in the code,
[16:05] but we can also share it in the Unity catalog in our descriptions on where we are getting this information from. Then, I have used the entity attribute value stream now quite
[16:17] often. We didn't have that in the beginning, but then we struggled with really this decoupling of the streaming sources and the batch sourcing and the different regions. So, really this enabled the
[16:30] decoupling, enabled us to that we don't care how often comes an update, we just put it into our change feed and then we get it, and that we have all the different sources in the same format.
[16:44] Then, when talking about the attributes, what which columns do we want to have um in the final table. Easiest would be to just keep all the different
[16:56] attributes, each service one column, each ECU parameter one column, but then 30,000 columns, and then you cannot cannot work with parquet tables anymore. And when you tried with Trino to access
[17:08] such a table, it fails more or less to understand the whole metadata because it's already too much context and too much in there. So, we see what can we combine, what also fits
[17:20] to the common queries because quite often the query for us is which services are active on this vehicle. And now it's just one column, one attribute that is needed to answer that question.
[17:36] And we are still struggling with some of the values, but kind of really combining some of them, but keeping the most important on their own. Then for us, the change history was important
[17:48] both for the analytics part and then also for the audit to understand what has changed on a vehicle, what correlates maybe with some issues that have suddenly occurred, but then also for the priority-based
[18:01] updates so that we can easily look up, hey, from which source did we get the current active value for a certain attribute? Is the new source that's coming in therefore higher priority or for higher
[18:13] quality? Or is it of lower priority and we can just ignore this change. Then we are using quite some useful features
[18:26] from Databricks. We have the Databricks asset bundles we're using um for all the deployments to the different environments, to the different regions. We are using the auto CDC feature or
[18:38] auto CDC from snapshot to automatically identify changes that have been in there. And we are also using Spark declarative pipelines for the first part, reading
[18:50] from the sources, reading the change feeds, putting that into our entity attribute value feed. But then at the same time, know the limitations of the features that are there. We tried to use the Spark
[19:02] declarative pipelines also for the rest of the pipeline in the beginning and we failed quite fast because we have a quite complicated merge logic. So, we are really only merging the attributes that have changed for the vehicles that
[19:15] have changed. So, in each micro batch, the merging might look a slightly different the merge command that that we do for for speed because it's we don't want to write all all columns at the same time, but only
[19:27] the relevant for the speed in there. And that's why why we couldn't use the Spark data rated pipelines there, but then we go and for the Spark streaming features and all of that.
[19:40] And with that, the time is over and I thank you for listening. I'll be here for some more minutes and I'm happy if you have any questions, comments, ideas what we could do better.
[19:53] I'm happy to talk to you and explain more about that. Thank you very much.

Learn more about the Databricks Data and AI platform.