From Millions to Billions: Scaling Geospatial Map Analytics with Databricks

Summary

• Geospatial Data at Scale: S&P Global leverages Databricks to process billions of geospatial data points for commodity market insights without separate geospatial databases.
• Performance Optimization: Strategic filtering and clustering of massive point datasets improves query performance while maintaining one set of geospatial-capable tables in the Databricks Data and AI platform.
• Hierarchical Indexing: H3's hierarchical hexagonal cells aggregate data on demand, scaling from neighborhood to global views and enabling efficient visualization at massive scale.

From Millions to Billions: Scaling Geospatial Map Analytics with Databricks

S&P Global and Databricks jointly hosted a webinar to dive deeper into how S&P Global has applied large-scale geospatial map analytics to derive Commodity Market Insights
You’ll learn how to:

• Reduce costs and complexity by eliminating the need for separate geospatial databases while maintaining one set of geospatial-capable tables in your Lake.
• Improve query performance by strategically filtering and clustering massive point datasets.
• Aggregate data on demand using H3’s hierarchical hexagonal cells to scale from neighborhood views to global contexts and visualize these at scale

Blog: https://community.databricks.com/t5/technical-blog/customer-blog-scaling-geospatial-analytics-efficient/ba-p/131234

Chapters

• 0:00 Introduction and agenda
• 2:09 Company overview and problem
• 4:49 Requirements and challenges
• 6:26 Databricks app demo
• 14:24 Spatial SQL and data types
• 17:53 Platform capabilities
• 20:30 Visualization options
• 23:06 Simplified architecture
• 25:55 Data engineering patterns
• 28:28 H3 indexing insights
• 31:29 Resources and blog
• 33:00 Future roadmap and Q&A

FAQs

How can I eliminate separate geospatial databases?

By using the Databricks Data and AI platform with geospatial capabilities, you can maintain one unified set of tables instead of separate databases.

What is H3 hexagonal indexing?

H3 is a hierarchical hexagonal indexing system that lets you aggregate geospatial data at different scales, from neighborhood to global views for flexible analysis.

geospatial analytics platform?

This video demonstrates scaling geospatial analytics from millions to billions of data points using the Databricks Data and AI platform. It showcases map visualization, spatial indexing, and distributed geospatial queries on massive datasets. The platform enables organizations to extract insights from location data previously impossible due to scale constraints.

How does Databricks improve geospatial query performance?

Strategic filtering and clustering of large geospatial datasets within Databricks improves query performance for massive-scale analytics.

What are the business applications for this approach?

S&P Global uses this approach to derive commodity market insights from geospatial data, demonstrating real-world value for market analysis and decision-making.

Full transcript

0:00 Introduction and agenda

Um welcome everyone to our uh joint webinar between data bicks and S&P global commodity insights. Um the topic here is from millions to billions scaling geospatial map analytics with data bricks. So uh that we'll tell you a story of a kind of real life story of things that uh we've done. Data bricks and S&P Global have done uh goodness I guess about year and a half or so is when I first started working on this uh related projects around this project we're talking about now.
Um so this goes back a little ways of just kind of a lot of um activity uh between our two organizations and we wrote a blog about all this that we have at the end of the presentation. will want to make sure that you have access to that and a GitHub repo as well. Uh but let's go ahead and get into our content today. Hit the wrong button there. Um yeah, so I'm geospatial product specialist at data bricks. Um in my role just uh briefly I focus on product and engineering a lot on the road map.
Great things that are happening there. We'll talk about a few of those things sort of in passing today. And then I also lead our SME group at data bricks where we um dig into various customer use cases and challenges and uh kind of put our heads together to help solve various problems. I'm going to hand it over to uh uh to give you the agenda and then uh go ahead and get into the content. >> Thanks. Hi everyone. I'm a geospatial engineer at S&P Global Commodity Insights. been here for around three years now and we're going to show you some of the geospatial things we we've all worked on here.
Um quick run through of the agenda. Um we're going to go through the problem statement and some really cool demos uh with. Uh we're going to do afterwards a deeper dive into an architecture including some data engineering and query patterns and uh what's to come from data bricks. Really quickly just some housekeeping. Please stay on mute uh during the recording. There will be a Q&A session at the end but uh you can ask your questions in the chat and through voice uh later on.
Next slide please. So a little bit about us about S&P global commodity insights. We offer a complete view of global energy and commodity markets to enable our customers to make decisions with conviction and create long-term sustainable value. We drive innovation leveraging data and valuable insights and commodities using cutting edge technology with robust data strategies for our clients. Next slide please. So a little bit about two large platforms inside of commodity insights.

2:09 Company overview and problem

Uh one data platform called advanced data workbench ADW and plat connect. Uh ADW is a very very large platform compris comprised of many tools and technologies built upon open source. We help any type of data citizen. So whether you're a data engineer, a data scientist, derive insights on the on their data. On the bottom, you're going to see this common capabilities uh which are basically common tools our users can use to derive insights uh for our clients.
There's you can think of this as like a common orchestration layer uh for example that can help you ingest, enrich and deliver data. Uh these common capabilities help our users do all types of those things. And one of the places that ADW delivers data is to our client-f facing platform called plat connect. In platon connect we have many many types of uh supported workflows uh including real-time market activity, personalized dashboards, AIdriven insights and advanced tools not limited to maps as as this is a geospatial talk.
Next slide please. Um a little bit about the scale of ADW. We have 75,000 daily jobs with 37,000 thus far tables onboarded terabytes of data processed for over a thousand active users. Next slide please. And so uh previously you know uh the our platform and inside of CI we required separate geospatial databases to enable uh map querying for our users spatial querying. Next slide. and uh we would ingest uh source data into ADW and essentially write it and read it back and forth into our platform between these geospatial databases to support this map querying.
Now, anytime you go ahead and write data somewhere, you're paying for the cost of storing that data there and also for transforming and all the compute costs. right now. Of course, uh this meant that for our clients, this meant that they would have less fresh data that they would have to see on the map. Next slide, please. And so, uh the ask that and Data Bricks helped us out with was clear but but challenging, right?
Three big points over here, right? First, we wanted to go ahead and see if we could simplify this data architecture such that we could remove all the redundancies of uh minimizing the storage and compute costs, right? And we needed no matter where you are on the map for our users to have performant queries to retrieve geospatial data. And on top of all our existing capabilities, we wanted to go ahead and see how can we provide more analytical capabilities, deeper insights that also scale.

4:49 Requirements and challenges

One big point here that's not on the slide is that we wanted to be able to do this for some of our most massive uh geospatial data sets including large time series. Uh next slide and over to you. >> Yeah. So with that uh introduction to you know what S&P Global was trying to do in this uh use case um I'm going to give you a little bit on the data brick side. Obviously, I won't get into everything, just kind of give you the overview of how we how we um did it.
Um and there's a a reference app that uh I will focus my attention on because um we're not really able to get into the actual, you know, S&P, you know, kind of commercial data and all that good stuff. So, um so this reference app is really where I'll focus my attention, uh which is, uh really how I worked with them, uh because I never was working with their data either to be clear. Um so on the data brick side um let's just look at some of the data engineering.
This is um really what you'll find also when you get to the repo and um the blog at the end here. Um a series of um uh things that we did steps that we took to prepare the data uh as far as like there's a journey of how how like things that didn't quite um provide the scale that they wanted and then there's sort of the final pass at the data engineering as to what um was the winning combination to meet their requirements.

6:26 Databricks app demo

Um so with the um with the notebook uh and the and sort of the preparation of the data that went into um to getting ready for the app. Uh I just want to draw your attention over here to this screenshot on the right side of the of the slide. Um and so we have this base data that was really the downloaded initial data uh for all the New York City taxi trips which is uh you know kind of the hello world of um of geospatial data.
Um but it worked well in this case for scale. Um and really you'll see two paths. One is a path um schema- wise both paths look the same. Uh but one path has less data. It has about 33 million uh pick up and drop off data points. That's really what we provided in the repo or access or kind of instructions for how to set it up for that level. Um because we don't want to overwhelm everyone to you know go to kind of a full billion plus scale uh data.
Um but the bottom one is that billion plus scale. It's actually 4. 7 billion uh pickup and drop offs. I'm going to show you both and sort of what we go through. Uh but basically in the same sort of steps of schema preparation, one does smaller data, one does bigger data. Um and then I just want to point out on the preparation of the data, we use data bricks built-in functions. will call out a couple of those uh to prepare the latl long data into um H3 cell ids uh using our built-in uh functions that support that on data bricks and then also uh we prepare some geometry data um which also allows us to use that as the basis of filtering.
So when uh queries are pushed down to data bricks data bricks can optimally uh find the right data that matches the viewport in the app. So um so that was you know just a light touch on what we did for data engineering and then on the unity catalog side you know for govern access of really any uh securable asset that you might have notebooks dashboards uh models uh tables of course and views are what most people think of uh but there's a a long list of things that we um actually help our customers secure um and so in this case I'm just showing you kind of a screenshot for those of you who may not have seen uh inside of data bricks um where the with that, let me go ahead and bounce out of the slides briefly and I'll jump over to the app.
Now this app is not meant to blow your socks off. It's really meant to make it real um kind of what the uh scale uh or the interactivity and what we're trying to solve for looks like. Um, so, so in this case we have, you can see the 33. 9 million uh, uh, rows of data. We're zoomed out in this case around the area. You can see there's a lot of bad data unless people really were um, getting dropped off in the middle of the Atlantic.
U, but we'll leave that to the side for the for a minute. And so what we have is, you know, this need for um, a lot of data that's summarized into um, H3 resolutions. So at different H3 resolutions, what's the what's the sort of aggregated uh count of events that might be filtered in different ways by different criteria etc. Time might be a factor of course for filtering. Uh and at our scale here with this size of data, we're at likeund 1.
5 million is like the deepest red here. Good old downtown uh Manhattan here. Uh so so let's go ahead and zoom in. So, as we zoom in, we'll watch our our scales uh change because we're actually using an interactive app. Now, nothing's happened yet, you may have noticed. Um, but when we hit refresh on the map, we basically set our view in this reference app. And, uh, that allows us to drive a query to the back end. And, um, that comes back with now, uh, a more fine grain H3 resolution that matches our current, uh, Zoom level uh, and matches the bounds on our app.
Um, and I think I mentioned it, but we're kind of down through this catalog, schema, table, column, uh, sort of navigation. So, we're in one particular place here. Um, I won't do it now. I might do it in a minute, but let me go ahead and go a little further on, uh, Zoom, and then I'll show you kind of how that um, schema navigation looks. But, let's go down just a little bit further, and we refresh the map again, and then we'll see um, a finer grained um, H3 resolution coming into the picture.
So really nothing too magical is happening here. In this case, what's happening with the app is um we've told it the column that has the H3 cell ids we want to work with. And uh what that's doing is then dynamically um using that and then going up or possibly down the um H3 hierarchy to figure out uh what's the right parent cell given the zoom level to uh to aggregate into. Um, so as we zoom, you know, here a little bit further.
We have, uh, 82,000 is like our deepest red. And let's go a little more, see what we get here. As we go kind of all the way, uh, more to the finest grain, uh, that this that this um, um, column supports. So, let's let that run its query and return. Now, this is all running on a data bricks app. um you know so it gets into that governance model of um your data kind of the app itself is run in a in a secure way and uh when we push all the way down you can kind of see how we've gotten down to this uh finer finer resolution and you can see all the pickups and such and kind of use your scales to help you understand.
Now this this is obviously a very generic example um but just to kind of um you know just to kind of show this you know this is in my catalog that I have um or the catalog I happen to be in and so whatever's in that catalog is really how we're navigating. So this is kind of like a d um a flexible explorer for H3. Uh not to u you know kind of totally belver the point but some of you might need to see this with 4. 7 billion.
So I'll just show that real quickly. So, I I went ahead and prepopulated with the my co-workers um data that has the full data uh so far with the um with the New York data. And we'll look here and we see that our scale is really on the order of like 150 million or greater in that darkest red. So, we're at a completely different scale than the 33 million. And let's go ahead and zoom in on that. Now, this is just uh Manhattan itself.
So, it doesn't really break outside into the bur burrows like the other one did. Um but uh we run this and we're getting kind of the same snappy performance. Data bricks is doing all the work. It's doing all these aggregations on the fly and uh we can kind of come in uh you know at various levels and see uh more and more data kind of aggregated into the H3 cell um information. Um so hopefully that um that helps make that a little more real as far as what the um what the data is doing, what the data engineering is doing for you.
Uh this kind of just puts that same picture on the screen for the slides after the fact to kind of uh make it clear, but also just to show in the repo um this reference app um is available and it basically has instructions and you can adjust you know things as you need to. But this is basically the YAML file with a little bit of uh Python command that you can uh send uh right in the um preparation of the data. you can go ahead and generate um the app the DAB uh data bricks asset bundle that um puts all that together.
So all this is sort of a reference for you to be able to use. Um so with that let me just keep going kind of kind of taking a step back above the actual um demo here and just give you a couple things about what we're doing with spatial SQL on data bricks. Um so um we've launched in the last couple of months uh we've launched public preview status or level for our spatial SQL um expressions and types. Um there's over 80 and a growing number of functions we support.

14:24 Spatial SQL and data types

These are spatial temporal ST type SQL functions with multil- language bindings. So you have the bindings for Python and Scala as well. Um, if you're familiar with kind of the Spark data frame world, you can tap into that side or just stay more in the SQL data warehousing world if that's where you're most comfortable. It really doesn't matter. Um, so we've we've we've launched that and um, basically we're getting into with that the ability to do, you know, you could imagine various con constructor support, various conversion to and from uh, interchange formats like WKT, WKB and GIOJSON.
Um there's um you know various measures you can do various uh spatial operations like uh like uh spatial join operations you can do things of that nature. So there's a growing you know kind of a core set of um what everyone uh our customers have been asking us for kind of as a as a main set of uh functionality. There's always a lot of things everyone wants and hopefully we'll get to all of them eventually. Um but that's sort of to introduce the SQL expressions themselves.
And then just also on the type side, it's I would say it's uh also very important that the types are here. Um so these are um this wonderful thing has happened uh that's culminated really this year starting in February with Parquet and Iceberg uh in particular. um some really key uh uh sort of um um uh you know partners or uh just organizations out in the community um have uh really come together to say it would be great for the community if we just had really good standard uh big data types for geometry and geography um spatial data and so th those decisions were approved back in February like I said um and so data bicks has been involved in those uh we've definitely are very very supportive of all the initiatives around that.
So these open data types are um are really uh coming along throughout this year. More and more systems are standardizing to them and with them. Data bricks is definitely uh in that uh camp of standardizing uh helping our customers standardize to them. We we sponsor Delta Lake which is another big data format and so Delta Lake is also getting the same sort of uh community standard support um as is um spark is uh in process to also be able to handle these same types.
So it's a really wonderful thing for the community to have all this come together. Data bricks are in as a product as a capability. Um, we offer our own built-in native uh geometry and geography types that are very much uh interoperable, shall we say, when when you need to run commands to import and export. Uh, but we're we're definitely handling uh kind of a data warehouse style uh type system to support uh you know scaled performance uh with our expressions that we support as well.
Um, so hopefully that makes sense. I I linked in the link to the geometry and geography docs. Um and before I linked I didn't say it but the links to the um uh 80 some odd expressions are linked on the slide prior as well. Um now with that we kind of hit you know kind of the high points here. Um we have a lot of customers using data bricks in some form or fashion to do geospatial. Geospatial is such a wide topic. Um you know they're doing all kinds of things. we'll say um we have our H3 expressions which is really what we've uh kind of uh emphasized the most with the visualization side to prepare for H3 for kind of scaled approximate style rendering.

17:53 Platform capabilities

Um you're not forced to use H3 but it's definitely a strong option for u good price performance shall we say the most uh kind of price performance you could probably find. Um and then uh we we have this everywhere. So our geospatial functions, you know, they're in our photon engine. So there's uh very performant uh kind of um API compatible with uh open source spark, but um actually rewritten for um kind of a vectorzed native execution uh uh that just drives a lot of great benefits uh for uh workloads.
Um and then it's available like in our notebooks. I showed you that kind of as a screenshot and in just straight up DBSQL, our SQL warehouses. Um, we're a flexible platform. As I kind of wrap up some of this, we're a flexible platform. Um, so, so you know, we really don't want you to think, oh, data bricks doesn't have every function I want, therefore I can't use data bicks. Um, really, um, we're we're very accustom uh, the SME group that I lead and and really all of our account teams, etc. , Everyone's very accustomed to um talking with um any customer, any potential customer about how to um work in uh their if they have library commitments or custom code that they've written kind of work with you uh to figure out what um is needed to incorporate data bricks plus whatever else you have in mind.
It might be a it might be a thirdparty integration of some sort as well. Um so we're very flexible in that way with a lot of uh great partners and um uh support a wide range of use cases. Um spatial data engineering uh these are more for your links after the fact. Um but we support uh you know a really serious production grade um ETL uh preparing your data for uh various use cases. Um we're very good at that uh for sure.
Um the Spark declarative pipelines are one way you can buy down maintenance and um just sort of make it simp uh add simplicity to building data pipelines. Um and uh the there's a lot of other things like blending um support for like iceberg with uh Delta Lake if if that's of interest to you that we support as well. Under one commit you can get both table formats written uh and managed together. Um so as I kind of uh wind out my my part here to introduce you a little bit to data bicks um you know when you think about visualization options because we are definitely showing you you know a app running on data bricks as a reference um you know I just want to point out there is a spectrum here and and every part of the spectrum matters um um the built-in part of data bricks is going to be closer to like a BI type experience um and you know we do have um kind of an ongoing oops we have an ongoing set of capabilities around you know how you can render things on a map in our AIBI dashboards.

20:30 Visualization options

Um and so if you're doing a chloroplast map or you're doing say a point map today and maybe you're adding lines and polygons and maybe some H3 in the future, you'll be able to render that in line with data bricks. Um, but apps are really kind of in that middle ground where uh it's not built into data bricks but you want to take advantage of some of our uh of our APIs uh more directly and kind of have more flexibility on the um interface that you're using.
Uh we recommend data bicks apps. What's shown on the screen here is um we have um you know kind of a dash uh you know python frontend engine that data bricks supports uh you know python frontends. We support javascript frontends like react etc. Um but this is this is an example of using Dash with data shader and hollow views uh to be able to build an interactive app that uh could be powered by data bricks. This happens to be from Dash directly.
Um just full disclosure but you know the types of apps that you can build on data bricks are kind of uh indefinite. You can build uh anything you can imagine and have the time or ability to do. Um, and then this one down here is another one that's like a um, uh, a built-on data bricks that blends a lot of context information beyond just spatial. It blends uh, more um, sort of incident reporting and uh, even calls that are coming into a call center to route first responders.
Um, so this would be another example of an app and the flexibility or power that you can um, that you can bring to bear, including bringing generative AI into these apps as well. On the far right here is really where you get into the most polished, the most sort of sophisticated experience for full GIS or geospatial uh map rendering. Uh and so we have a range of partners. Um I don't want to call out any because I don't want to leave any out to be honest, but um I recommend that you look at our technology partners and um we can definitely help you um kind of match up uh kind of where you're at and what you're looking for with what data bricks can uh support uh with partners or builtin or apps.
Um so with that let me pass it on to uh continue. >> Sure. Thanks. So thanks to spatial SQL and the spatial types what this meant for us uh inside of CI inside of commodity insights is that we would be able to ingest our source data into ADW as before. And next slide please drum roll and query directly um in inside of our platform and inside of ADW. Uh next slide please. We're going to have a t a little bit of a deeper look into what uh the architecture that we had and how you know uh spatial SQL was able to simplify our architecture.

23:06 Simplified architecture

Here you're seeing a typical ETL architecture, right? We would extract data from our source systems uh transform it such as this New York City taxi example where we would select different geometries and load it into our external uh geospatial databases where every single point would become a certain geometry. You can get the next slide from me. And so what would happen here is uh you know we would create different map layers for for our users and we would have one geospatial field uh which was a limitation with a specific formats uh for our map layers.
Now this makes sense in some places but if you take a look back to our source table you know in some cases we have extremely extremely wide tables many hundreds hundreds and hundreds of columns right and so if you follow this gray arrow here to the gray columns if one of our users wanted to go filter on any column that was not in that source table as from that map layer from the right side of your screen they would have to exit the map select the extra columns they would want to filter on uh where our APIs would go ahead and create temporary tables our users would would filter on.
Right? Next slide please. And so this is what we were talking about before where we end up having this whole right side from this before architecture diagram where you would have data duplication that again you would have to end up paying for the ETL cost for storing and and moving the data there for the compute cost and it was just less fresh data compared to where the source was. Whereas now afterwards right we are able thanks to spatial SQL and the spatial types keep our table with all the existing columns as they were right no matter how wide our tables were right and just store it directly into in Delta Lake right and now uh we could have our map layers come uh dynamically as needed from that source table um through SQL serverless warehouse and so basically we would just move all that complexity into the API layer while maintaining performance, right?
Next slide, please. And so this helped us uh unlock a lot of things, right? Firstly, it simplified dramatically as you saw in the last slide our our data architecture well while keeping the cost low. Uh so this meant uh you know we could have multiple map layers, right? Such as following you know that uh taxi pickup and drop off example where we would have a pickoff point and a drop off point in a single table. um as different map layers from the same source, right?

25:55 Data engineering patterns

And this meant, right, we would be able to visualize uh in and other tools um related geometries much easier. So following to the third example on the screen here on on the right side, we would be able to make decisions easier such as if we follow this example, we could say, hey uh you know if we were picked up in let's say the around the S&P office uh in New York City by water street and we were rushing to the airport in to JFK, right?
Uh, we could go ahead and filter on pickup points with around S&P, drop off points around JFK, and filter on the where we could around the office get picked up to make our flight and find the columns and then and the areas in the H3 cells such that we would have the lowest travel times, right? And there's, you know, this is just one of the many analytical flows that we could have here. But it's really just to show the pattern of, you know, not only simplifying the data architecture, but what it can provide to um our other data citizens inside ADW.
Uh next slide please. Some uh data engineering here and query patterns, right? So firstly, uh you know, we have all the spatial functions, right? So we could take you know a longitude and latitude that represented a pickup point or drop off as following this example. specify the SR ID um of that geome uh geometry and you know get the native uh uh delta geometries the geospatial types uh we would go ahead and query directly from that table with thanks to spatial SQL how this query looks like and you know there's tons of built-in optimizations here but you know you can use one of the many other spatial uh functions such as st intersects to say hey if I'm if our users map users are using the map and want data at this part of the screen such as the yellow bounding box here, the yellow rectangle that you see.
We would just say, hey, given this uh yellow bounding box, return me all the points that intersect uh that intersect the bounding box. Uh next slide, please. The problem, however, was if that if our users would go ahead and zoom out too much on on the map viewport right there, you know, that's a lot more points that that query has to run. So this is where we ran into you know uh uh performance challenges and how we solved them was using the built-in H3 APIs right um H3 has many sizes right many H3 resolutions which you'll see in the next slide that follows but think about about it this way multiple H3 resolutions that cover the earth that provide these kind of analytics friendly columns that you will soon see to help discover trends right next slide please so here a little bit deeper look into the data engineering and query patterns over there right we have this uh one of those H3 uh APIs that uh was mentioning the long lat as H3 where we could provide for example a pickup point long lat given specifying a specific H3 resolution out of the 16 uh resolutions from H3 uh and in this case we indexed our pickup point on cell 8 on resolutions 8 10 and 12 and uh prepare that data in our table.

28:28 H3 indexing insights

Now, towards the right side of the screen, we would have a two-part query, right? It's it's one query, but just follow us through here. Using other H3 APIs uh in a longer subquery that we'll have referenced later in in in the slide deck, we have basically functions that go ahead and cover your map viewport with H3 cells. So, that top right image that you see on the screen at any H3 size that you want, right? Uh and then afterwards once we know all the big integers representing those H3 cells that cover our map viewport we could go ahead and do aggregation functions such as counting let's say how many points in our for the H3 cells that are in the map viewport um that are in every single H3 cell right so I could go ahead and count hey how many pickups are in every single H3 cell and this unlocked these kind of big insights for us that we didn't have before right such as saying hey how many pickup drop off points are in every single H3 cell.
And you can see here on the on the bottom image if if for example an H3 cell had more pickup points than another H3 cell, you know, we could color it yellow as opposed to dark purple, right? And the real unlock here for us was that hey, you know, you could do something like, hey, count how many points are inside of every single H3 cell, but you know, this is just a group by query. So what this meant for us is that we could go ahead and you know use any function um uh using any column that's allowed.
So you could go ahead and go back to that H3 taxi example and say hey give me the H3 cell uh give me all the H3 cells in this map viewport and tell me the lowest uh H3 cells that have the areas that have the lowest travel times. Right? Kind of going back to that rush to airport example. So, you know, you could really go wild here and pick out whatever type of analytical needs to get insights that you need. Next slide, please. >> Cool.
And this is going to go back over to you, where we're going to drop uh and then mention some of our references. >> Yeah. So, this is uh just a couple more slides left here. Um this is the blog that uh a number of people contributed to. uh is the main uh contributor with a number of others from data bricks and S&P uh commodity insights contributing. Um but this summarizes kind of the core um learnings and uh final technique uh that we've talked about today.

31:29 Resources and blog

Uh I think it's a good one just to be familiar with for um you know kind of the power of H3 and aggregation matched up with an interactive app that you're looking to possibly develop. Um so the blog is linked here. Um, and then the repo is also linked on this slide. And um, I'm just going to, if I can get out of the slides, that is uh, let me try that again. Uh, if I if I can, I'm just going to click on the repo just to pull it up.
So, this is a public re repo um, GitHub repository. And, uh, inside of that, you'll find basically all the assets that are tied to what I showed today. Uh, minus that kind of full scale 4. 7 billion scale. uh it talks about how you can do it, but you'd have to work a little harder to get the full scale um prepared. Uh but hopefully this becomes a good uh you know, a good resource for you um if you're interested in this topic, which if you're here, you you very well might be.
Um so that's uh that's really the last thing before just a little bit on the horizon for data bricks. Um so we're in this public preview phase with our types and um our our SQL functions themselves. uh this public preview phase will take us into, you know, the early part of next year. We expect to move past that into general availability. Those dates are always very fluid uh until they're not. Um and so uh hopefully watch early next year for some more information on that.

33:00 Future roadmap and Q&A

Um but as we kind of get on that uh that kind of drivers towards general availability and even beyond uh one thing that I can say that's coming up uh uh should be out this month even um with a blog that we're working on as well uh is around join performance. So just uh some things that we've done beyond uh what we did with S&P Global and and up to this point uh built into the product that allow uh uh even better scaled uh join performance.
So um that's an ongoing you know area of work. Uh but hopefully another bite at the apple is happening um uh soon here that we'll that we'll be able to announce. Um other things just briefly u there's work on the JDBC driver. Um that work is well underway. Um, so I expect more to come from that uh soon, very soon. Um, but you know that'll be the ability for the driver to really be able to properly handle these new types that we've talked about.
Um, also I mentioned the Spark uh spatial type, the SPIP that went in the proposal for these spatial types for Spark. Um, that work is very far along. Um it's uh if you go to the link here, you'll see that it's um has a good chance of making it into the Spark 4. 1 release, which is basically in a release candidate phase. Uh it's moving towards that now. Um so we're getting very close possibly to see those announced as part of the 4.
1 or maybe soon after. Uh you know, you know how these things go. It's it's a little bit of a um not not a guarantee on everything. We have more work on enhancing our geog geography support. Um what we worked on with S&P Global was more focused around geometries. Um but there will be uh increased support for geography expression handling. Um where today we have say 100% geometry and maybe 25% geography. Um we're continuing to work on bringing geography all the way up to match uh geometry support.
Um and that's a that's a great area of u of investment that we have. The V3 specification for iceberg I just mentioned briefly for the community types. Um that is what we are working on in data bricks sort of widely for all the v3 capabilities but uh specifically geospatial uh support uh in our um platform uh is underway um so just keep an eye out for when that's actually available. Delta uniform kind of goes with that uh our support to manage iceberg and delta lake uh in the same transactional commits.
Um and then lakebase interop our transactional database um there's work there um that is uh also well underway um that um will allow lakehouse and lakebase to um to be more readily synchronized for spatial data. Um so with that uh I think we still have time here. Let me um let me pause and uh also uh you know we want to open up for any questions if you have any and then just want to say you know I know some people may not ask everything they want here or you may want to just connect uh offline in some way.
Uh, so we've put our LinkedIn information here if you want to connect with us on LinkedIn um and or uh message uh either of us, both of us, however you choose to do that. um feel free to um reach out and you know we're happy to talk more about your specific

Learn more about the Databricks Data and AI platform.