Accelerating Autonomous Driving: Geospatial Intelligence at Scale with Databricks

Summary

Rivian migrated legacy geospatial workflows from PostgreSQL to Databricks as a design partner, optimizing spatial joins across 450 billion records of autonomous driving data enriched with HERE maps, OpenStreetMap, and Overture road attributes.
H3 cell system optimization initially reduced spatial query time from 37 seconds to 5 seconds, and adoption of DBR 17.1's native geometry and geography data types achieved an additional 10 percent performance improvement while eliminating complex H3 pre-processing pipelines.
Pandas UDF-based event detection enables 3D lane merge and split identification for Rivian's large driving model, with AI agents accelerating pipeline development across the autonomy platform's analytics, LDM training, and reinforcement learning applications.

Accelerating Autonomous Driving: Geospatial Intelligence at Scale with Databricks

Rivian's autonomy platform processes millions of vehicle logs daily to build the large driving model powering hands-free driving features. Working closely with Databricks as a design partner, Rivian migrated legacy geospatial workflows from PostgreSQL to Databricks, optimizing spatial joins on 450 billion records of autonomous driving data enriched with HERE maps, OpenStreetMap, and Overture road attributes. Starting with H3 cell system optimization that reduced query time from 37 seconds to 5 seconds, Rivian adopted DBR 17.1's native geometry and geography data types, enabling 10 percent faster spatial joins while eliminating complex H3 pre-processing. The team built Pandas UDF-based event detection for 3D lane merge and split identification, accelerated pipeline development with AI agents, and created a seamless data pipeline that identifies road attributes, lane curvature, and driving conditions for analytics, LDM training, and reinforcement learning applications.
🤝

Chapters

FAQs

How does Rivian use Databricks for autonomous driving data?

Rivian's autonomy platform processes millions of vehicle logs daily, using Databricks to perform spatial joins and geospatial enrichment across 450 billion records of autonomous driving data. The platform integrates road attributes from HERE maps, OpenStreetMap, and Overture to support analytics, large driving model training, and reinforcement learning applications.

What is the Rivian and Databricks design partner relationship?

Rivian served as a design partner for Databricks' spatial SQL product development, working closely with the Databricks geospatial product team to validate real-world workflows and provide feedback as native geometry and geography data types were developed. This collaboration influenced Databricks' spatial capabilities while giving Rivian early access to features optimized for their production pipelines.

How did Rivian optimize geospatial queries on Databricks?

Rivian first optimized queries using the H3 cell system, which reduced spatial query execution time from 37 seconds to 5 seconds for their large-scale joins. They then adopted DBR 17.1's native geometry and geography data types, achieving an additional 10 percent performance improvement while eliminating the complex H3 pre-processing steps previously required.

What is the role of Pandas UDFs in Rivian's geospatial pipeline?

Rivian uses Pandas UDF-based event detection within Databricks to perform 3D geospatial analysis, specifically identifying lane merge and split events for autonomous driving scenarios. This approach enables the team to process complex multi-dimensional road geometry at scale, generating the structured event labels needed for large driving model training and reinforcement learning.

Full transcript

[00:09] All right, this is I think the last uh time slot for talks for the day, so let's try to pick it up, make it through. Um an important question to ask before we get started, who here owns a Rivian? Okay, you come right up to the front,
[00:25] you get a prize. No, just kidding, just kidding. to own a Rivian? Yeah, who wants to own a Rivian? Exactly, hey. Um my name's Ken Martin. I'm the product manager for geospatial at Databricks. Um I have a small part to play here,
[00:41] really just to introduce Chinmay and Philip, uh who I've worked with uh over the last year and a half um as the they've developed their solutions and we've built things in product in Databricks. Uh so Philip and and Chinmay asked me to come and introduce and provide some of
[00:57] the context for our collaboration. Uh so I said, "Yeah, sure." So I'll be up here for a few minutes and then I'll hand things over uh to these guys. So, uh you know, the big question when you're on the product team at Databricks like I
[01:13] am is how do you know what customers want? How do you know what to build? How do you know what will work in the real world at real world scale with real world SLAs, you know, all the things, right? Um and at Databricks you know, we try to find
[01:29] uh customers uh who we refer to as design partners, right? People who will uh dig in deep, kind of iterate quickly, and provide a feedback loop as we incrementally improve the product. Um but the benefit to the customer or the
[01:45] design partner in this case is you know, they get to validate their real workflows and influence the things that we're building, right? Uh so with Rivian, you know, we really found a great design partner. Uh and I'll say, so I I live in Victoria, British Columbia. Um, and and Philip here is in Belgrade
[02:02] in Serbia. And we have a shared Slack channel. And so one day I was like, you know, Philip, how about I come over there for an hour, fly around the world, and we'll have a meeting in person and kind of kick things off. Uh, and Philip, a man of few words, just gave me an emoji.
[02:18] Good. So I did. Um, and we, you know, we had an on-site, and from that point forward, uh, this was sort of in the process of our private preview for spatial SQL, adding geometry and geography data types,
[02:33] working on efficient spatial joins. Uh, you know, Philip was sort of a an eager partner at that time to to dig in. Um, over the course of months, you know, we could reach out to the team and say, "Hey, like, can we test this against a real pipeline? Can you guys evaluate
[02:49] this? We've changed something." And and and really look at the performance. Uh, so we had like this really good kind of partnership. And uh, I'll pick on Philip a little bit more here. You know, Philip immediately found all our problems or errors. Um, and of course, you can see he's like, "I
[03:04] fixed this. I fixed this myself." A true data engineer. Um, so yeah, it was really good to have this kind of collaborative interactive feedback with the team. Um, and this took place over the course of, I would say, 6-7 months, where we would give them custom builds, they would give us
[03:19] feedback, and we move forward to public preview, uh, later in the fall with spatial SQL. So, uh, just a summary of of the kinds of things that, uh, Philip and team, and I've also aggregated, uh, some of the
[03:34] feedback we've got from customers in the automotive space. Uh, all the things that the feedback we received, these are now all a part of our product. Uh, so if it came to new expressions, like, uh, azimuth and make point and these kinds of things, some of them, you know, fairly, you know, basic standard kind of
[03:51] spatial operations, uh but we've tried to really focus on the things that were needed and then improve the performance where, you know, it was a challenge, a bottleneck, or some point of friction. So, whether it was uh computing, you know, lines using make line, uh the
[04:06] Boolean set operators here like ST_Union, ST_Difference, ST_Intersection. Um and of course with spatial joins, we've really spent, you know, some time to make sure that these are the most performant operations um that we could deliver in the product. And I think uh I won't steal any of
[04:23] Philip's thunder here, but um I know he's going to show some results which show that, you know, it really dramatically improved the efficiency of the workloads that that they were running at Rivian. Um I'd also say we received feedback on AI BI maps, which now supports rendering
[04:38] geometries and geographies, so you can do that in dashboards. Um also within Genie, um you know, just understanding the experiences that, you know, the teams at at Rivian were were trying to uh leverage all of the new capabilities um as we've rolled out new spatial
[04:54] capabilities to make sure all that's working. And then uh we released Spatial SQL as GA last week, uh which includes uh um Iceberg V3 support, Delta sharing, and uh is based on top of the the open
[05:09] table format specifications uh that our team has uh you know, worked on and contributed to. The last slide here, you know, these things are all kind of spatial spatial specific things. Um but a good design partner is trying to use the platform, complete workloads end-to-end, whether
[05:25] it's BI using the engine, um of course on the governance and real-time side of things. You know, we received a ton of feedback from the folks at Rivian across a number of teams, not just Philip and Chinmay, but but other teams that are kind of spread across the world. Um so, really just want to say thank you and
[05:41] hand things over to the guys, and they're going to talk through, yeah, how it how uh the geospatial um you know, feedback they gave us turned into a really good productive things for them. Okay? All right. Here you go, guys. Thanks a lot, Kent. And really appreciate the
[05:57] Yeah. Cool. Uh We'll We'll We'll first try to understand what kind of data do we collect from the Rivian vehicles. Uh starting uh R1 Gen 2, all Rivian
[06:15] vehicles come equipped with 10 external facing high-resolution cameras, five radars, and starting with R2 launching this year, will have a front-facing uh la la f- f- like front-facing lidar.
[06:34] With this wide variety of sensors, we are able to run early fusion on board. Um we are able to perform a m- a multi-model
[06:49] a like tr- like trajectory on board. Using this data, we have built something called as the Rivian autonomy flywheel. We already spoke about the sensors on board. The next part comes the autonomy data
[07:07] recorder. This is the piece which tells when and what kind of logs to collect. This is adding a lot of smart logging to our on-board sensors. Uh think about like, you know, in in
[07:24] like some kind of an interaction with an agent. All this data is being logged by the autonomy recorder. This data is being uploaded to Rivian cloud in a highly privacy-compliant way. Next comes the really important and
[07:42] critical piece of this flywheel, which is the large driving model. Think of this as a close cousin of some of the LLMs or large language models. It uses similar state-of-the-art neural net-based architecture
[07:58] like transfer like transformer-based encoding, uh auto-regressive prediction, and reinforcement-based like learning. But now, instead of the tokens being words, these are actually driving
[08:14] behaviors which are fed into this model. As our fleet size grows, uh the model gets better. And these improvements are being pushed to our customers every month as part of the
[08:31] over-the-air updates. Before we push every update to our customer, uh we carefully vet our software by running it through a large set of simulation scenarios.
[08:54] Uh let's talk through some components of the Rivian autonomy platform. As we said, we are processing millions of vehicle logs per day. As these logs are landing on cloud, we are ingesting these logs into several stores of data.
[09:10] The primary use case of this data is for training and simulation. But we also have a use case where we want to run data mining and analytics on top of this data.
[09:27] Think of this as we have a lot of data, but how do we find useful data from it? That's where a lot of this metadata enrichments like geo geo tags, road tags, lane attributions comes into play.
[09:45] Now, Philip will talk in depth about some of the geospatial work work which we have done on the team. Thanks, Chinmay. So, it was around 7 years ago
[10:01] when I first got into contact with with geospatial data. At the time, I was working in telecommunications industry. We were designing fiber optic network infrastructure.
[10:16] The job was interesting, but there was this one part that was repetitive. We were using QGIS to select uh infrastructure objects on the map. And depending on design size, this could
[10:31] take a while. At the time, I was also reading a book about algorithms. And I stumbled upon something called Dijkstra's routing algorithm.
[10:46] They provided a full code example in C. And I started thinking, "What I'm seeing right now in QGIS map are essentially graphs. They have their start and end node and their distance. So, in theory, I could build and refactor the C code.
[11:02] I have a script that is going to select start and end node and all the shortest path objects in between." So, I I go ahead and do that. It was my pet project for around 2 weeks. At some point, I get the algorithm
[11:18] working. I translated it from C into Python. It was Python 2 at the time. And I was really happy. It was working exactly as I wanted it to. So, our director one day comes up to our
[11:33] office and I say like, "Come here. I want to show you something. Would you give me time to develop this tool for the team cuz I think it would eliminate a lot of this repetitive stuff." It turns out such a tool already existed in QGIS as a plugin.
[11:49] But, whether he liked the effort or he liked the project, I don't know. Um it wasn't very long before he promoted me. Fast forward to today at Rivian, I once again have the opportunity to work with geospatial data
[12:04] at much larger scale and on some very interesting projects. Folks, thank you all for joining our talk. My name is Philip. I'm part of Chinmay's team and with around 10 people, we form the data platform team here at Rivian.
[12:20] We're not geospatial engineers. We do lots of interesting things like batching, streaming, interacting with different kinds of databases, and even a little bit of DevOps. Geospatial is only a small but very interesting part of our overall
[12:35] projects. Today, we're going to talk about several things. We're going to talk about how our journey with Databricks geospatial started and we're going to present a series of projects each bigger in
[12:50] complexity and scale compared to previous one. We're also going to talk about how we tested one of the latest DBR releases, 17.1 at the time, which introduced significant performance improvements in spatial joins.
[13:06] And finally, we're going to present a project that we are currently working on, which contains geospatial 3D problems that were very interesting to solve and also we used Pandas UDF vectorized framework. So, how these things usually start is
[13:23] with a legacy project migration. I get approached one day with a question, "Hey Philip, can we make this query from Postgres work on Databricks?" I start looking into documentation and I find there is quite a bit of support for
[13:39] geospatial SD functions on Databricks. I ingest Postgres data. I refactor the query syntax a little bit. And what you're seeing now on the screen is literally our first geospatial query on Databricks.
[13:55] Back to the project. What was the architecture we wanted to migrate from? It was AWS batch job that interacted with Postgres DB. We wanted to move geospatial queries to Databricks and some IO operations to
[14:14] So Chinmay and I rework all the connection classes and allow seamless integration with existing code. But there was an issue. This was not scaling well. On large data sets, geospatial queries could not finish successfully.
[14:32] And at the time, Kent was sending us these monthly updates about the geospatial product. And in one of those, I find something called a hybrid join. Where Databricks recommends users to use an H3 cell system
[14:48] to collocate geometries to the same general area and only after run more expensive geospatial operations. All right, cool. What's H3? H3 is an interesting library developed
[15:04] in open source by Uber in 2018 with full support on Databricks. It divides the entire earth into these hexagons of different resolutions and there is a parent-child relationship between the hexagons of different sizes.
[15:21] We built a small POC that integrates this H3 cell system and we get got great scaling results back. We were able to reduce single query processing time for from around 37
[15:37] seconds to around 5 seconds, which was great result for us at the time. And this is how it looks like in action. I would often use Kepler.gl for visualization. And there are some interesting functions
[15:54] there like the K ring, which could be used for nearest neighbor analysis. And tessellate is pretty cool one as well. You should check it out. And there are more than 70 H3 functions um in Databricks.
[16:12] Okay. Let me grab a sip of water. Okay. So now that we were confident that we can build
[16:27] scalable geospatial pipelines, the next logical step was to enrich vast amounts of data that we had in our systems with different road attributes. We started by ingesting three different data sets.
[16:42] HERE, which is high definition polygon-based data source of US highways, and then Overture and OSM. Overture is like a mastered version of OSM data set, which contains less attributes, but
[16:58] should be of a higher quality. And we use the OSM as a fallback in case we ever need to get some attributes not present in Overture. So we built a series of these data pipelines that enrich our internal data sets.
[17:13] And some of the use cases for these data sets are Obviously, analytics. We could analyze autonomous driving features usage based on road attributes. We could curate data sets. And we could feed these curated data
[17:30] sets into an LDM model by providing it uh examples of specific road conditions. Okay. This next project was a different data set all together.
[17:45] And was much larger in scale. At the time, more than than 450 billion records and more than 23 terabytes of data.
[18:03] The request came from analytics team. We were very interested in analyzing autonomous driving features usage, specifically on highways. And they came with a request that we provide the output table in the same format. We just add the road
[18:20] attributes column to the end. So, we had the constraints we we had to work in. Then we engage in series of enrichment priorities. Since we are interested in highways,
[18:36] let's go with the high definition map first. And all the others will come after that. And the main challenge here was really high probability of scale. And also, we had to use different H3 resolutions for different data sets.
[18:53] And this is how it looks like as the intersection for different geometry types. On the left, we can see that it's very easy to join a point to a polygon. But when we need to join a point to a
[19:09] line string, this is where it becomes difficult. In the cases that we were interested in, HD feature maps did not cover the entire US highway roads, so we would want these overture road attributes
[19:25] there. But when the vehicle was in the outer lanes of a very wide highway, we couldn't really catch the line string, which was in the middle of the road. We considered some approaches like nearest neighbor, buffering, map
[19:40] snap. We quickly discarded nearest neighbor and map snap because it would require a significant code refactoring. So, we would go with buffering approach.
[20:01] we would draw these series of concentric circles around vehicle uh point. Where the goal was to capture these line strings that would be in the middle of a highway very wide highway road.
[20:16] We would also prioritize geometries captured with the smallest buffer available. In that way, we would limit some unnecessary geometries with wider buffers. This approach would guarantee us
[20:32] that we would capture more highway information than than by using HD maps alone. So, let's get back to the original problem. We had around 140 million time stamps per day.
[20:48] We could just do 140 million spatial joins, which is what we tried. But this would run very slow, much slower than we would want. So, we considered some other approaches. What if we could trade the performance
[21:05] of a spatial join for the performance of a regular join on clustered columns. We go ahead with this idea. In each batch, we select 30 We select distinct coordinates.
[21:20] Which cuts down number of coordinate enrichment from 140 to around 30 million. Much less. We store this and later connect them to the timestamp level data.
[21:35] This was fine. But also creates a problem. Our pipeline was incremental in nature. Which means we would not process the same coordinates twice.
[21:56] we would end up missing subsequent routes and they would have no route information. To To tackle this we added something called lookback processing.
[22:12] With this, we would just go to our output table, see which coordinates had their road attributes missing and fetch them from our already enriched table. Even with the introduction of a lookback process, this was still faster than running so many spatial joins.
[22:33] Again, core idea was that we we would trade the performance of a spatial join for regular join on clustered columns. Okay. So, one day Can't Reach Us Out to Us and says "Hey guys, do you want to test out this new DBR which has improvements in
[22:49] spatial joins?" And we're like, "Yeah, let's let's do that." This would introduce storing of geometry and geography data types in a Delta table and better uh spatial joins.
[23:05] We would engage in scale testing and analyzing these results. Just for reference, on the top image, blue column, is pipeline runtime without any H3 optimization in
[23:21] So, this is just to paint a picture how important H3 was in the entire picture. All right. So, to test 17.1 properly, we need a baseline. We capture that using 16.4
[23:37] with H3 applied. And then, we perform uh two tests. One was we would not yet store any geometry into a Delta table. We would just basically switch DBRs.
[23:55] And the new DBR didn't really like that. Its performance was slightly slower, which points to a fact probably that there was some change in how data type is stored in memory or processed in memory. But, that's fine. The last test was what's really
[24:11] important. We would store these geometry types into Delta table and then run spatial joins only on that without any conversion in in between. And this was really important. This was a major milestone. We would get 10%
[24:26] better uh improvement performance on smaller batch sizes and roughly the same on other batch sizes. But, think about what this solves. We could now drop our H3 tables and just use the
[24:43] join on geometry without any anything else. Just how much effort it takes to optimize uh the pipe spatial pipelines with H3. You have to choose resolution size, build the arrays, explode the arrays, and cluster the table. This significantly significantly improves all
[25:00] of that. Finally, we will talk about this project that's very interesting. It is geospatial in 3D and contains some very interesting event detection logic inside.
[25:20] It's another migration project, you won't be surprised. We had this legacy C++ code that interacts with an MBTiles file, which is an SQL SQLite library with geospatial data. And we wanted to migrate this to
[25:37] Databricks and use Pandas UDF and Spark for distributed processing. To do that, it would be good if we can have a baseline or a ground truth. So, we use Bazel to build a Python wheel
[25:54] with the hopes of running this on Databricks as well. It turns out Python wheel has a very specific Python dependence, but also architecture. We couldn't run it on Graviton processors. We had to use x86.
[26:10] And we had to go all the way down to DBR to find the the Python version that supports it. All right, we put all of this together. We run the compute and somehow this thing runs. It's very slow, very inefficient, and
[26:26] the DBR that we chose goes out of support very soon. So, it's one more reason why we needed to do this migration. So, with a ground truth or baseline, we proceeded to ingest road information, lane information, and we built
[26:43] parent-child relationships for road segments and lane lane segments as well. This is very critical if you want to detect any lane splits or lane merge events.
[26:59] Okay, so let's think about what we have. We have our data. We have the schemas. We have ground truth and we have the original source code. And one more thing is we don't have C++ skill.
[27:15] So, this is a perfect task to feed into an AI. We spawn some agents and we and we give them a goal. Translate the C++ library into a Pandas UDF. And after around 10 or 11 iterations, this ends up working.
[27:31] I call it partial success. Some tags were created okay. Some needed slight adjustments and some were very far off. But I wasn't going to give up. I engage a few in few more iterations hoping to fix the issues,
[27:48] but I see the model losing contact and it was draining too much of my energy just to steer it in the right direction. At this point, I consider whether it's worth to just do it myself. Another reason is the code was getting really complex.
[28:04] 1,600 lines of Pandas UDF code. There had to be at least one engineer in the team who understands that. So, I go in. I fix some of the logic changes for the events. And we end up with the working version of lane merge attribute tags.
[28:22] Here in the videos, we show that this is a 3D problem. It was interesting to tackle. And there are some um successful detections of split and merge and some on ramp events.
[28:40] Now, before I hand over to Chinmay, who's going to talk about some future projects, I do want to say one thing. You all work in different companies, different industries even. And I'm sure you all have this legacy project lying around somewhere that management is anxious to touch because
[28:55] it's very important. But I want you to take it upon yourself and put the extra effort to build that POC that shows that you can migrate your legacy infrastructure to the new and modern stack. If you're using geospatial, okay, you
[29:13] can use Databricks for that. If it's something else, then use something else. But I want you to give it a try and have your own Dijkstra's routing algorithm project. And even if it's not entirely successful, you never know. Someone may just like
[29:28] it. Thank you. Awesome. Great job there, Philip. And the first part of the story was really great. Really
[29:45] enjoyed it. Cool. So, let's look at some of the forward-looking projects, right? So, uh Rivian recently announced uh some of the hands-free features which will be uh released to our customers
[30:02] towards the end of the year. A critical piece of this hands-free features is the map being really correct, right? So, we are working with our internal maps team to fix some of the inconsistencies map by repeated readings of some of this
[30:19] wide variety of like the sensor information. Lane attribute enrichment is something which Philip spoke about, uh but this is going to be like an ongoing activity where we we will add like more tags in the future like
[30:36] uh lane changes like uh what is the lane curvature and like what not, right? And then uh Rivian recently announced a partnership with Uber. Uh we have ambitions to deploy 50,000
[30:53] robot taxis in several urban areas. Uh but it's not not hard to imagine some of the geospatial features getting used there. With that, uh thank you for being a part of this presentation and I'll open it up
[31:10] for the questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.