Spatial SQL on Databricks: Geometry Types, Performance Benchmarks and Use Cases

Summary

Native geometry: Spatial SQL GA brings geospatial analysis to the lakehouse under Unity Catalog governance.
File-skipping optimization: native geometry enables statistics and improved spatial joins like PostGIS.
Map visualization: DeckGL, Kepler and partners (Esri, Felt, Carto) enable AI-built site selection.

Spatial SQL on Databricks: Geometry Types, Performance Benchmarks and Use Cases

Talk by Kent Marten, Staff Product Manager, Databricks
Spatial SQL on Databricks adds a native geometry data type to Databricks SQL, so geospatial analysis runs on the lakehouse under Unity Catalog governance instead of in a separate GIS stack. This video covers what reached general availability, why a native geometry type matters for statistics and file skipping, what improved for spatial joins since public preview, how the roughly 100 supported expressions compare with PostGIS, and benchmark results against Databricks Runtime, plus map visualization in AI/BI dashboards and an AI-built site selection analysis.
Learn why the native geometry data type matters (statistics and file skipping), what improved for spatial joins between public preview and GA, and how the roughly 100 supported expressions compare with PostGIS. See benchmark results against Databricks Runtime, map visualization in AI/BI dashboards and Databricks Apps with libraries like DeckGL and Kepler, and partner integrations with Esri, Felt and Carto. A demo builds a site-selection analysis with Claude and Genie Code, and the talk explains when to choose Lakebase with PostGIS versus Databricks SQL. It closes with the roadmap, including geography support and geometry and geography in Spark 4.2.

Chapters

FAQs

Can I run geospatial queries on Databricks?

Yes—Spatial SQL now has general availability with a native geometry data type on Databricks SQL. It supports roughly 100 geospatial expressions similar to PostGIS and runs on the lakehouse under Unity Catalog governance.

How does the native geometry type help performance?

Native geometry enables statistics and file skipping, so spatial queries read less data. Spatial joins also improved significantly since public preview.

Can I visualize maps in Databricks?

Yes—map visualization in AI/BI dashboards and Databricks Apps works with libraries like DeckGL and Kepler, and partner integrations with Esri, Felt and Carto add advanced mapping.

Can I build site-selection analysis with AI?

Yes—the demo uses Claude and Genie Code to build a site-selection analysis on spatial data, showing how AI can help reason about geography and locations.

Full transcript

`00:00` Introduction: spatial data on Databricks

Yeah, let's just dive right in. My name's Kent Martin. I'm the product manager for Geospatial at Databricks. I've been there for about five years. This is a lightning talk, so we're going to try to cover a lot of content in a little amount of time. Happy to take questions afterwards and meet with folks as well. It's super loud? Pause. Audio guy. Hey, he's saying it's too loud. Is it still too loud? Is it too loud? Yeah, I don't know.
Maybe it's just one headset? It's loud for you too? You can change the volume on the headset. Hey this doesn't count against my time, reset. How about now? It's okay for people? Gentlemen in the front, how's it sound? Can you hear me? Okay. Am I too loud? Am I just right? Okay, let's try to go from here. Just a quick recap on kind of Lake House philosophy, right? We talk about simplifying. You know, there was a day not too long ago when you worked with warehousing and data lakes and they were separated, which means there was a separation of use cases, a separation of governance, and basically an opportunity to duplicate data, things you don't want to do.

`01:52` Why the lakehouse is the foundation for spatial data

So the Lakehouse paradigm emerged from this to address these problems, right? So with Lakehouse, you get all these things solved with unified governance, Unity Catalog, one platform. All use cases can be run. You can now do spatial analytics at massive scale, which was a challenge for lots of folks. And of course, working with structured or unstructured data, you know, that's the benefit of the data lake in the Lakehouse, right?
So we're all good with Lakehouse. This is a simplified architecture for working with all types of data and certainly applies to working with spatial data. So why spatial now? Why did everyone show up here? I'll say just a couple things. We see broad adoption of a lake house pattern. We see that almost every organization has spatial data. That's a bit of the cliched thing, but I can share at Databricks, over 70% of our customers have run spatial workloads.

`02:28` Why spatial now?

So it's not like a myth, it's like a real thing, right? People are working with location data, super important. And then as everyone knows, the agentic future is now the present. So you need to have the right foundations if you're going to run AI workloads and have agents that can reason about the world. They need access to spatial data and that sort of foundational components. So at Databricks, one of the big projects that we've worked on in terms of building the right foundation for vector spatial data has been Spatial SQL.
Spatial SQL is like a post-GIS kind of equivalent offering. It's now GA as of last week. And of course, this effort started a while back with the addition of geometry and geography in Parquet, Delta, and Iceberg, which is the foundation for this offering. So now we're GA. Great. I want to give a little update on what was improved between our public preview and our GA, just so everyone kind of comes up to speed. So everything on this slide, I'll say, has come from a customer request or a customer use case that's been shared.

`03:38` Spatial SQL GA: what improved since public preview

So we focused a lot on expressions and sort of building out capabilities. If you think about the expressions that we support, there's about 100 now. If you compare against post-GIS, there might be like 700. So we're not going to go build all those things. But we listen to customers. And if there are use cases you're trying to bring to Databricks, please let me know and we can add new expressions. On the platform side, again, you can use Spatial SQL in declarative pipelines, data warehousing, in notebooks, with serverless, wherever you need to.
It's all available. So if you went and used the latest, it's there today. I will say we've really focused a lot on performance. You know, performance is really the differentiator when you're talking about kind of common expressions and standard expressions. So when it comes to spatial joins, you know, between public preview and now, we've run this against real-world workloads. I gave a talk with Rivian the other day where we talked about some of their spatial data processing, and they saw dramatic improvements when working with spatial joins.
We have lots of feedback from customers that's really been validating. These are numbers on the screen. I would just say, if you have workloads that execute spatial SQL, go and try it. If you're working with this kind of spatial data, don't store well-known text, don't store well-known binary in columns anymore. Use the geometry data type, use the real native stuff, because you get statistics, you get file skipping, you get all the optimizations that come with those types.

`05:32` Native geometry type: statistics and file skipping

So we put a lot of effort into performance, and we'll continue to do so. And then just from an ecosystem perspective, again, I mentioned we have alignment across the OpenTable formats. Our team has been contributing to that effort from the get-go, so we're really involved in that effort. And we now also support Delta sharing, which I just want to highlight for those of you who are building data products, sharing data products. now DeltaShare can support the geometry and geography data type.

`06:12` Spatial SQL performance benchmarks

So that's ready to use. So just, you know, I'm going to harp on the performance piece, but I wanted to share some of the benchmarking that we've done. This is a comparison against our own Databricks runtime at Public Preview. So even between Public Preview and now, when we looked at the spatial benchmarks that were released by the Apache Sedona community, we saw dramatic improvements across a broad set of the queries and those were the ones that we kind of focused on like query two was already you know 80 times faster than Sedona so we ran a lot of you know all these these kind of queries also against Snowflake and other you know competitors and I'll just say like go ahead and run it yourself the numbers are really good it really depends on the workload but we've seen especially with massive scale data really dramatic performance when working with spatial data compared to competitors as well.
But, you know, try it out yourself, man. You'll find out. Okay, so performance has always been an important thing that we've worked on. Visualization is another area where folks, like I've met here at the conference, lots of folks who've said, well, what about building maps? What about sharing maps, dashboarding? So let me just give a little bit of a tour of what's possible. So with built-in Databricks AIBI dashboards, there's always been point maps and core plet maps.

`07:32` Mapping in AI/BI dashboards and Databricks Apps

So you could map administrative data like countries and states and provinces kinds of things. But now that supports geometry and geography. And what that does, it just really opens up, you know, kind of the world, you can map anything now, right? You can model anything as spatial data, you can map it in AIBI. With Databricks apps, you just have a lot more flexibility in terms of building a purpose-built application. You can leverage visualization libraries, whether it's DeckGL, Kepler, these common libraries within the app framework.
So yeah, again, if you're building something that has maybe a more specific purpose, this is a path forward. And then we partner with lots of folks, both partners, like technology partners, and then of course I mentioned some of the libraries you can use. There are others, so there's always like a third-party option as well. I know some of the partners are in the expo hall, you know, stop by, check it out. And yeah, this is just kind of teeing up the partner side.

`08:40` Partners: Esri, Felt and Carto

We work a lot with the folks at Esri. Who here uses Esri within their organization, right? Yeah, I mean, just a ton of folks. So a few things. There's already connectors in some of the Esri products, like ArcGIS data pipelines. We've collaborated with the Esri team to support custom data feeds that allow you to serve data directly from Databricks. And then there's also the ArcGIS Pro integration that's planned, hopefully will come later this year.
With Felt and Carto, you have kind of cloud-native GIS organizations, and they're building kind of interesting MCP. The Felt MCP is now available in the Databricks marketplace, which is great, and Carto's building agents, which you can use and run on top of Databricks as well. These are things I definitely encourage you to explore. Lots of other integrations. Anyone here use QGIS? We have an official plugin now. You can find it for the man in the back.
Okay, let me do a short demo just to dig into some of the stuff we're talking about. So I became obsessed with golf because I was sick of losing to my friends. So this was my idea for a brief demo at the summit, which is, can I just build a site selection? where would I place a golf simulator in San Francisco so I could come and practice when I visit the city? This is a kind of workflow that's like multi-step if you're starting from scratch.

`09:59` Demo: AI-built site selection analysis

You have to pull data, demographics, segmentation. You have to know the road network and maybe have walkability or drive time. There's a lot of components that come to it. So I'll ask you this question. is anyone using, you know, Genie code or Claude code to like write their pipelines now? So about half the folks here, like I'll say like once you go in, you just never go back, right? So for this kind of analysis where I generated like a 30 cell notebook, I'll show you the prompt that I used.
And when I write these kinds of things, like I write a really detailed prompt because I knew in some cases, you know I know where the data is so I you know I share that I know I'm going to use census data I'll get the API key right like Claude can't go get an API key for me but I'll show you my prompt just as an example like I give really explicit instructions in terms of where I'm starting and what my outcome is that I need and I would just say like I get really good results when I've done this although I've never you know golf analogy here like I've never hit the hole in one and just like got the pipeline to run day one.
But I ran this in both Claude and in Genie code. And again, this is data from scratch, right? Like I'm pulling data from APIs. It's not data that I already have in the lake. And I'll say there were probably about five or six improvements I had to make along the way. Like I couldn't get it to run initially. But this took 35 minutes, which I thought was just amazing compared to how I might have done this previously, right?
So this is the notebook. I'll just kind of flash a few of the things. You can see I'm just calling outside APIs, the census data, census track, road data, isochrones from Valhalla. So yeah, I mean, these agents and these AI tools are just so efficient and proficient. This is sort of like the route to go. So I ran this analysis, site selection in San Francisco. Let me show you what it looks like in an AIBI dashboard now that we support geometry and geography.
So here's the table that I ended up creating. And I'll just have you focus on the layer column here. So you can see, and I'll just talk about this for a second, AI BI dashboards today support one layer of data on a map. You know, we got to work on that. It's in our roadmap. So we plan to support multiple layers. But right now, I've actually unioned the data set that I built in the pipeline so that I could use filters to sort of toggle layers.
And of course, you know, here's my geometry data over here. I can go to my site selection view. I've kind of pre-canned some of this because I knew we'd have audio problems. so I would have not too much time. But you can see, maybe you can see that. Right down here in the bottom right, there's a new option to choose custom geometry. I've already chosen that. I need to choose my geometry field. And now I can see my actual geometries rendered on the map.
In this case, I'm looking at the intersections of the demographic data that I pulled. You know, I could also look at the competitors and see their walk times and the overlap for sort of like the catchment area around where I want to place these things. So, yeah, lots of sort of the standard interactivity that you get with AIBI, and this is an area that we're looking to improve. Right, so I just wanted to make sure everyone could see this, go try it out, and now it's easy to share your own analysis.
I'll highlight just a couple things. you know we have solution accelerators for site selection it's worth experimenting with you know you can pull you know these assets it could include code notebooks you know different approaches lots of ways to go here and you know these accelerators come with like templates for building databricks apps so you can build really sophisticated elegant applications i won't dive too deep on this today and then i just wanted to show one other thing which is a little bit of the art of the possible.
We talk about vector data here, spatial SQL for now, but with Databricks apps, you know, you have a ton of flexibility. So I have a colleague who lives in that house, this farmhouse, and he has one telephone pole and he said, you know what, I work a lot with utilities customers and energy customers and they need to go inspect poles. They need to see vegetation encroachment, right? Like an interesting use case. And usually they'll go and fly a drone or they'll have LIDAR capture to go and monitor this stuff.
So what he did was, you know, he did that. He flew a drone. He captured, you know, the arrows of the, the blue arrows show the direction the drone was looking. I'll just select some of the cells where the images were captured. I said, render here in this application. But he basically took the images, you know, nerfed them, generated a point cloud, trained the point cloud with a SAM3 model, and labeled some of the features so that I could go ask questions and develop a 3D scene that I could actually interrogate.
So here you can see some of the house. I can see my 3D feature. I've got, you know, embedded Genie. How tall is the pole? You know, I can ask simple questions. Why didn't I shut off my Slack? You know, gosh, every time. I'm in the middle of it. He's probably over there. Jeannie's thinking. I mean, I hope it takes just a second. Sometimes it takes a little bit for some of these apps. Okay, yeah, so it can respond and give me a reasonable answer, right?
It's a 21-foot, 7-meter pole. Great. So, yeah, what I'm trying to show here is, I'll go back to the slides. You know, that's not like a LIDAR replacement, right? But I can see a way in which this could augment a workflow where I could easily go and work with drone imagery and build like an application that allows me to go and inspect, you know, the real physical world, right? I know lots of folks are working with that kind of data.
Happy to learn more about what you're doing. Okay, let's jump back here to the slides. I just have a few minutes left, so I want to talk about a couple of things. So in the last year or so, Lakebase has really become a prominent part of the platform. So Lakebase has managed Postgres on Databricks. It comes with all the standard extensions, including PostGIS, PG Raster. There's other ones involved. LakeBase is like a pillar for doing spatial analysis on the platform, right?

`17:14` Lakebase with PostGIS for spatial applications

So I work on like the LakeHouse team, but also work with all the sort of teams that care about spatial data. If you're migrating from Postgres, PostGIS, you know, this is an easy path. But really, you know, the important thing to think about here is what's the purpose of your application, like what's the data serving. And of course, we're talking about, is it a transactional application, or is it a big analytical application?
So I just have a slide that kind of talks about the decision tree between the two. And on the left, for LakeBase, you can just really think about low latency, high concurrency type applications. You want to serve something real time, this might be the way to go. On the other side, if you think about Lakehouse, DB SQL, I mean, almost everything there is a synonym for big data, more complex joins and aggregations. You want to use Lakehouse DB SQL, because it's a powerhouse for all that stuff.
So definitely, just think about your use case and the purpose. So this week, we also introduced Lakehouse Realtime. So it's like a little wrinkle in the offerings. And while Lakehouse Realtime today doesn't support spatial SQL that we've built, it will in about one or two months. So my team, this is our priority to get Lakehouse Realtime completely compatible with the same queries you run on Lakehouse DB SQL. And then, of course, they also mentioned LTAP during the keynotes, which is like a compute isolated way of calling the lake-based files that you have.
So directly from object storage, you can use Lakehouse real-time and basically just point the engine to where the data is. So there's a lot of flexibility working between these three sort of engines. And yeah, again, just consider the use case. But I want to highlight Lakehouse real-time because the performance, the concurrency are quite incredible. And as we deliver geospatial onto that platform, I expect we're going to get another kind of step function improvement in terms of low latency spatial queries.
So exciting times. Okay. I have five seconds on this clock. Take a picture. No, I'm kidding. They'll let me go a little bit. So yeah, our priority is real-time Lakehouse. If you use H3, it's already supported in real-time Lakehouse. So you can get started today if you're involved in the private preview of the warehouse. I mentioned geometry a lot. The geography data type we also need to complete. It doesn't yet work with some of the standard expressions you would expect.

`20:21` Roadmap: geography type and Spark 4.2

So that's also an important sort of milestone for us. And then when it comes to open source, you know, we're trying our best to accelerate the contributions to open source for Spark with respect to spatial. And geometry and geography are going to be in Spark 4. 2 this summer. So that's an exciting milestone. There are some of the expressions that we built. They're not all in the open source yet, but ones that allow you to convert between, you know, binary and, you know, the common interchange formats. so that you can create these data types.
Last thing I'll mention here, like raster support native in the product. Like who's working with raster data? Yeah, a few folks are doing raster data, and I've talked with a few folks here at the conference. If you have requirements, use cases you'd like to talk about, because of course you can be successful today, right? You can use Sedona, there are other raster libraries. We have some Databricks Labs efforts that are sort of focused on that. where we're looking to bundle GDOL capabilities and make it easier for folks to work with GDOL APIs.
But on the note of providing feedback and engaging with me and the team that works on Geospatial, I can say, you know, in our recent blog where we announced GA, I'll just jump down here. I did put a little link in there. and that the response to that forum goes right to me so if you have a request if you have uh you know a use case or you want to meet with the team and talk about something uh you know what feel free to reach out uh we'd love to be engaged on that level okay so with that uh i'll just bring up the oh i have one more slide um there were a few talks here at the conference from customers They should be available on demand.
I definitely would recommend, you know, both of them, depending on, you know, your industry and use case. The S&P Global one does a little bit more like, hey, how do I involve agents? How do I build the right kind of spatial agentic workflow? So check it out. Hubert, that's for you, buddy. That's the guy from S&P. Get him. And then, yeah, that's all I got today. I'll be available after the session, you know, for an hour or so if people want to chat or you want to talk later on, just reach out to me.
So thanks everyone for making it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.