Analyzing Geospatial Data at Scale in Databricks for Environment & Agriculture

Summary

• Geospatial Analysis at Scale: Databricks enables horizontally scalable geospatial analysis using Delta Lake and H3 integration for vector and raster data.
• Environmental and Agricultural Applications: The video demonstrates real-world use cases in environmental monitoring and agriculture addressing climate change and resource management.
• End-to-End Processing: Covers ingestion, transformation, and analysis through production of geospatial data products accessible via standard GIS tools.

Analyzing Geospatial Data at Scale in Databricks for Environment & Agriculture

Talk by Luke Teacy, Associate Director - AI & Data, Deloitte (Sponsor Speaker)
Analyzing geospatial data has become a cornerstone of tackling many of today’s pressing challenges from climate change to resource management. However, storing and processing such data can be complex and hard to scale using common GIS packages. This talk explores how Deloitte and Databricks enable horizontally scalable geospatial analysis using delta lake, H3 integration and support for geospatial vector and raster data. We demonstrate how we have leveraged these capabilities for real-world applications in environmental monitoring and agriculture. In doing so, we cover end-to-end processing from ingestion, transformation and analysis to production of geospatial data products accessible by scientists and decision makers through standard GIS tools.
Here's more to explore:
The best data warehouse is a lakehouse: https://www.databricks.com/product/databricks-sql
How to migrate legacy data warehouses: https://www.databricks.com/resources/ebook/migrate-your-legacy-data-warehouse-databricks
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

Why is geospatial analysis important for agriculture and environment?

Global population is predicted to peak around 10 billion this century, and food production may need to increase by 50% by mid-century. Geospatial analysis helps optimize land use and resource management.

What are vector and raster data?

Vector analysis in geospatial context involves points, polygons, and lines, while raster data is imagery-based data used for visual analysis.

What tools enable scalable geospatial analysis in this video?

Databricks, Delta Lake, and H3 integration provide the foundation for horizontally scalable geospatial analysis supporting both vector and raster data.

What is Deloitte's role in this video?

Deloitte collaborates with Databricks to enable scalable geospatial analysis with real-world applications demonstrated for environmental monitoring and agriculture.

Full transcript

All right, I think we're about time. So, uh I'll make a start. So, just to uh introduce myself. So, I'm uh. I'm an associate director in Deote based in the UK. And today I want to give an introduction to analyzing geospatial data at scale using data bricks uh with uh use cases in environment and agriculture. So this work is inspired by work that we our team have done uh in collaboration with some of our partners and um and clients back home.
Um if you're not familiar with uh geospatial analysis then uh don't worry we'll be introducing the key concepts and etc as we we go along. Um but if you are then hopefully we still have some interesting food for thought in terms of how we approach some of these problems. Okay. So um just to give a brief outline. So we're going to start off uh with motivation and use cases. So why is geospatial um analysis important when it comes to environment agriculture?
Uh we'll then look at sort of architecture and tools. So how data bricks fits in uh with the rest of the sort of geospatial um platforms and tools and ecosystem. Uh and then based off of that then we'll go and have a look at uh specific problems and techniques around the use of uh vector analysis and raster analysis. So uh vector analysis in the context of geospatial data that's points and polygons and lines. Raster data is more when you're you're looking at uh imagery essentially.
Um so uh why is this important? So um it's no surprise I guess that uh you know some of the biggest challenges that we're uh facing today as a a society are the twin problems of uh increasing population and uh climate change and environmental protection. So just to illustrate that um so the UN currently predict that we're looking at uh a global population peaking this century around 10 billion people um and with that then we need to feed ourselves.
So you know by the middle of the century um that some predictions say that we we could need to um increase our food production and agricultural output by something like 50%. Um, and this comes at a time when we're already using onethird of all available land mass on the planet for agriculture. Uh, and it's not that we can just go and take another third, right? So, onethird of that is is forest, the other third of that is is desert and uninhabitable.
So, um, we kind of have to work with within the bounds of what we've we've got there. Um and then it should be no surprise then given given the extent of that uh you know the current estimate is that something like you know 13 to 21% of uh greenhouse gas emissions can be attributed to how we use the land so agriculture forestry etc to feed ourselves and provide the other resources we need. Um so uh absolutely very important area um and um yeah in terms of geospatial data you know so it's it's really about trying to say okay how do we gather all of this information um and really understand how are we using the land and and inform policy uh in terms of how we uh address these these twin challenges.
Uh so in particular just to give you a flavor of some of the use cases that we're looking at um so uh when we're looking at environment so one example is uh watershed and flood risk. So trying to understand using things like LAR data um can we model um you know what happens when we have increased rainfall and uh flooding in the future. How do we what sort of interventions can we make um in order to uh address that? Uh precision agriculture.
So helping farmers by you know analyzing the the soil and how we're using the land and can we target for instance you know how we're using resources such as fertilizer and pesticides etc in a more targeted way. um so that uh we can you know increase production but with less less input uh and less of a an environmental impact from things like runoff of fertilizer etc off the land. Um other things we're interested in is uh animal disease control and epidemiology.
So thinking about modeling how um disease things like uh bovine uh TB uh spreads and when we have an outbreak you know can we look at uh doing analysis around you know where are all the farms in the area how do we how do we start to control that outbreak etc. Um and then both within agriculture and environment you know basically looking at mapping different vegetation zones and how the land is being used. So uh for example if you look at the UK or the EU um we provide a lot of um area based schemes where we're um giving grants to farmers based on the total area of land that they they farm for different purposes.
So having accurate mapping um of different land parcels where the fields are etc. Uh really important for making sure that uh that farmers are adequately and accurately um given the payment that they need. Um and then looking at habitat. So some really interesting examples of that are things like uh blue carbon. So um you get environments such as salt marsh um and seaggrass etc. Um and also on the on land um pete land as well.
So these are all environments that really p um they really punch above their weight in terms of the amount of carbon that they can uh put into the ground. Um so trying to restore these habitats uh map and understand the different vegetation zones within them. Um you know these are you know really interesting uh challenges for policy makers etc trying to um uh improve those environments. Okay. So in terms of the data that we bring to bear so as I sort of mentioned earlier uh there's that comes in two main forms.
So we have uh vector data and this is sort of lines, polygons, uh points etc that you tend to see on on your average map or map application. Um and that's really efficient for doing lots of sort of analysis and statistics over over different uh pieces of land infrastructure etc. U but in terms of raw data quite often that comes in the form of imagery. So not just you know you might think of you know your typical uh RGB images etc but uh where you know each pixel is your your true color your red green blue etc.
Um but also with um satellite imagery etc. you can have uh multiband so you'll have uh near infrared data you'll have um lighter sort of digital elevation data. In that case the pixel values are you know representing you know different values. So in the case of LAR that'll be you know each pixel value is actually your height above sea level or or to some reference point. Um with near infrared as well. So if you combine that with um sort of visible uh light spectrum so you can do lots of interesting things.
So you can um calculate indexes that um tell you something about uh vegetation and uh water for flooding and water bodies etc. Okay. Um so luckily these things all come with their own open standards that are are well supported across different geospatial packages. uh and these we can leverage within data bricks as well. So in the case of vector data, it's typically things like giojson. Um there's also well-known text, well-known binary um which are probably not well known outside of geospatial circles, but um well known when it comes to GIS uh platforms.
So these basically provide a way of encoding your your points, your polygons, your lines etc. um within standard data types. Um and then what comes with raster data? So while that can use sort of uh standards that are based on standard um sort of image formats like like uh TIFF for instance um there are some differences. So as well as the image the pixel data itself you also have to keep in mind the metadata. So the actual mapping of you know where each pixel has to align to a particular geospatial location on the ground.
So that metadata has to come with it. Uh and the other interesting thing is obviously the scale. So it's not just single images. it's it's tile. So, you know, you're breaking up the entire globe or or region of interest into individual tiles. And quite often, particularly if you're you're visualizing this data, you're you you don't want to if I if you think about go um Google Earth for instance, if I'm looking at the entire planet, I'm not loading every single tile of pixel data into uh memory so that I can I can view that, right?
So, being able to view the data and analyze the data at different scales is important. And rather than computing that on the on the fly, we tend to store that as a as an image pyramid. So basically take the same data and we store it at different resolutions uh so that we can access it at different scales when we need to. So that's what the data looks like. That's the sorts of problems we want to solve. So what what are the tools that we can bring to bear um to solve that then?
Um so so when we bring data bricks into the picture the goal is not to replace sort of the other geospatial tools um that we have uh in our ecosystem. uh the goal very much is to complement that. So bring to uh bring to bear the scale and the versatility of data bricks to do large complex analysis, right? But very much to complement uh the sort of mature um features and um uh functionality that we have in in existing geospatial platforms.
Um so if we kind of start on the left hand side of the slide uh in terms of uh ingestion so typically the kind of data sets that we're bringing in we're bringing that in uh it might be filebased raers from large sort of aerial surveys that we've commissioned uh it might be calling APIs out to um satellite imagery providers um and bringing data both from sort of geospatially enabled platforms such as RJS Enterprise but also nonspatial uh databases as well um to so that we can bring all of this together and enrich it.
Um and while currently we're we tend to do this in sort of file-based uh batch ingestion workloads, there are some interesting use cases we're looking at where you we're doing more event driven um uh ingestion as well. So a nice example of that is uh in ARS um they support versioning of uh of map data. So for instance, if we're um if we're making changes to you know official land boundaries etc. all of that has to go through quality control and reviews before that those changes get committed to the main map.
So um uh so EZRI basically provide that versioning history if you like if all the changes have been made. Um so we're looking about how we can tap directly into that change history and drive like a a change data capture sort of um workflow in terms of bringing that data in. Um and that means that rather than have to reprocess absolutely everything on a regular basis, we can we can make take those changes in and make small changes to the uh our uh data products out the other out the other side.
And then in terms of how we're storing it, so I mean all of this still fits within the sort of standard medallion infrastructure. So, uh, bringing our raw data into to bronze, processing that through silver and building data products out the far end. Um, so all of our sort of vector data can go in there. Um, but sometimes with the raster data, we're still storing um, you know, raster files and sort of standard formats there as well.
But the same same principles and sort of governance, etc. still apply. Uh, and then we're pushing that back out the other end. So we're making those um uh insights available um by pushing the data back into some of these geospatial platforms such as ArcJS where you know we've got mature visualization um uh capabilities and you can integrate that with desktop GIS tools as well. Yes. Yeah. So we you can do both. So we we do store it as um so I mentioned the well-known sort of um for um text and JJSON etc.
But um as you'll see there sort of there are standard spark libraries that provide sort of like um custom uh geospatial data types. Um so you can sort of interpret that natively as well. Um but yeah but I think there's it's a growth area for data bricks as well. So they do have some built-in support for some of these things as well. Yeah. Um okay. So that's that's how we get the data in, that's how we get the data out.
Um but the other key thing um is making sure that the the scientists, the decision makers and actually the general public, you know, can discover the outputs of this and and make use of that. Um and it was interesting. I was at attending one of the talks I think earlier today or or or yesterday where they mentioned this as well where you know quite often within a large organization you could have multiple people basically coming up with the same analysis and not being aware that you know some of this analysis has already been done or the data was already available.
Um so in order to solve that um it's not specific to um geatial but I think it's it's quite an important one. Um, we've built uh sort of a a business userfriendly uh data portal. Um, and the way that works essentially is we uh harvest all of our sort of technical uh metadata for data assets um from both from Unity catalog but also from other systems within the ecosystem. Uh we then bring all of that data uh along with sort of the data product descriptions, the business metadata about what the data is, who owns it, um what it's for, how you access it.
Uh and we master all of that in in Microsoft Perview. Um and then we use data bricks pipelines then to take that u metadata out of purview, populate a a search index that then we can uh make available uh through data portals both to the um the public but also um uh internally as well. Um and a key sort of design decision we made there was to align the sort of metadata schema that we used for that to something called DCAT.
Uh Decat is a uh W3C um standard essentially that's used by a lot of um open data platform such as uh CCAN which sits behind some a lot of the sort of UK gov um data portals. Um so by aligning to that sort of schema it means that we can then uh actually make that data harvestable and export it out into um other open data platforms as well. Okay, so that's our architecture and in terms of some of the tools we're using.
Um, so before we get to sort of uh you know large sort of spark distributed libraries, it's worth calling out there there are a lot of uh sort of non non-spark native libraries and tools um that um are very useful and kind of provide the foundation for that. So uh GDAL uh sits under underneath quite a lot of um geospatial software. It's basically your your Swiss army knife of all of the um con IO essentially for different formats to do with vector and raster.
Um we've then got things like Shapely and Geopandas that um you know really quite mature in terms of uh handling this sort of data in pure Python. Um, raster io is is again our sort of main go-to uh non-spark library for doing analysis and IO with um raster data um etc. And then we have some other niche tools as well things like pi shed etc. So some other open data um sorry open source python libraries that we use for um analysis such as looking at um water watershed and um uh how water flows through the landscape.
And then when it comes to making that distributed u bringing spark to bear. So um couple of things we use there. So we use the built-in h3 support within data bricks um which mentioned some of the talks earlier in the week. Um so this allows us to do efficient indexing um using you know completely out of the box with data bricks. Um we use uh Mosaic which is a uh data bricks labs project um that has number of uh features on uh with vector and raster formats.
A couple other ones such as Apache Sedona uh we're looking at as well. Um but a lot of these libraries pay play nice with the previous non-Spark libraries that we mentioned on the um on the other page. uh and quite often actually we we can achieve a lot of what we want to do just by using uh userdefined functions with some of those earlier libraries. All right. Right. So if we look at vector analysis or lines and polygons um so um if we think about you know some examples of what we might want to do with that.
So it might be okay we've got a disease outbreak you know can we look at all of the farms within a certain radius of that. Um can we look at you know different habitats and vegetation zones within a a given region of interest. Um so again the sorts of common operations you want to do there you know it's things like you know aggregation or statistics or um looking at um maybe set functions over uh different polygons. So one example might say if you've got all the polygons for all of the fields within a farm you might want to join those together to have a representation of all the different farms within a given area.
Um so all of these things have broad support in all of the libraries that we we mentioned earlier. Um but probably the most interesting one in terms of computational complexity is when we're trying to do spatial join. So we have you know a large data set of uh land parcels or whatever it might be in one side. We have a region of interest or something else that we need to join or intersect that with on the other side.
Um how do we how do we do that in an efficient way? So this is where uh H3 comes in which I said is is out of the box um with data bricks. So H3 is an indexing system hierarchal one that um was first developed by Uber to um deal with some of the large data challenges that they had. Um and basically what it does is it breaks the entire globe up into these uh mostly hexagonal uh areas. uh and then you can basically take the cell ID for those different hexagonal areas to say right well if I have a set of polygons or points or geometries within the same cell and I can I can join on the cell ID.
Um so this is sort of what it looks like. So depending on the resolution of the uh data that you're looking at, you can go all the way from uh resolution zero, which is you know thousands of kilometers and in area um down to cells that are roughly about you know one one square meter. So you know you can take it down to quite a granular level. So it's really just about picking the scale that's best suited to the data you're looking at.
Um and then if we want to do a join. So say we have okay so I've got my land parcels and I'm trying to you know find all the the land parcels within a certain area of interest. U the first step is to take my different polygons. I map that onto the whole set of um uh H3 cells essentially that fully cover that um uh that polygon. Um and then that gives me a set of um uh cell ids essentially that are associated with that polygon.
Um and if I do that for both sets of geometries that are of interest um then I can uh basically you know use something like liquid clustering to do the join in normal way um between the tables. Um and that essentially gets you close close to the answer. Now obviously it's not perfect. So you know you can have you could have two polygons that don't intersect but still intersect the same cell. So you know there might be a bit of um uh refinement you need to do after that but uh but it gets you to where you need to be essentially um and you can make significant significant gains sorry um in terms of um computational time.
Okay so in terms of raster analysis again you know uh wide support for a lot of the basic sort of operations you might want to do. So things like um raster io so reading those sort of standard formats um such as geotiffs etc bringing that into um the platform uh the sorts of operations we might want to do. So you might want to do uh algebra essentially. So like can we do mathematical operations over uh different tiles different data sets that represent different um factors.
Um we might want to do statistics over those things as well. also taking the all of the tiles that sit within a certain bounding box and calculating uh statistics over the top of that. And there's also more complex things we might want to do. So I mentioned waterheds earlier etc. So trying to understand how water flows over the environment based on uh LAR data etc. Um so all of this sort of stuff we can we can bring to bear.
Um and just to give a uh one example of where we can use this sorts of analysis. So uh this is a piece of work we did uh to look at mapping intertital zones um around the coast. Um so to do that we were using uh sentinel 2 uh satellite imagery uh alongside 1 m resolution uh lighter um digital elevation model. Um and what we did was we basically take you can take a a time series of um uh satellite imagery over a given region of interest. um you can then take those that satellite imagery and calculate what we call a normalized difference water index.
So that basically is a very simple calculation on your I think it's a green channel and your near infrared channel which gives you a good indication of how much water or you know if there is water present at that pixel location. Um so we can do that for every tile in in the time series. Um then with uh certainly you can also use uh you basically treat like cloud cover as as missing values. Um and if you basically then sort of average you know the uh water index um over time for a a particular area what that basically gives you is an indication of how what percentage of the time um is that particular location covered by water or not.
So if it's out to sea it's going to be covered by water all the time. If it's in land, you're never going to see any water there, hopefully. Um, but if it's in the intertital zone, you it tends to change depending on um, you know, if it's high tide or or low tide. Um, and then we can refine that by doing sort of um, tidal in inundation modeling um, with the with the uh, LAR data. Um, and it gives us a, you know, a rough empirical um, measure then of understanding where these intertitle zones are.
Okay. Um so um the other big area for raster data is if we're trying to do uh computer vision. So number of different applications for that. So that could be everything from sort of object detection where we're trying to understand and identify um rural infrastructure etc. Um it could be uh quite often it's it's segmentation. So trying to you know look at the orthophotography and identify different vegetation zones or different areas of land use within the uh environment.
Um so again one example of that is is salt marsh where we're trying to map out the extent of the different vegetation zones in there. Um and the other thing that we mentioned earlier was sort of the boundary detection. So understanding where the boundaries are between fields and doing accurate mapping of um of farms etc. So various different uh models that we can use for that. So we've we've done a lot of work with uh EU net which is a sort of deep learning uh convolutional neural net um which was originally um designed I think for medically medical imagery but uh has you know reasonable success when it comes to um satellite data etc as well. um things like uh YOLO which is sort of more or less state-of-the-art in terms of um object detection and instance segmentation.
So um that's a a good candidate as well. Um particularly it's quite efficient in terms of you know you're not going over the image multiple times. It's it's quite quick in terms of its analysis. So uh that's also a good candidate when we're we're looking at trying to scale this um based off yolo as well. So recent paper uh that came out I think it was last year was uh delineate anything uh resolution agnostic uh field boundary delineation in satellite imagery.
So that's uh something we haven't explored yet but it's one we're quite interested in looking at. Um it's it shows really promising results in terms of um identifying those boundaries between fields at different scales. Um and other things that we've looked at in the past as well is sort of weekly supervised um techniques as well. So if you think quite often, you know, the challenge with um computer vision or any machine learning is getting that ground truth, getting that training data.
Um and you know, it can be quite a lot of effort if you're trying to go around and annotate exactly where all the boundaries are um in your different vegetation zones by hand and then use that to train your model. Um so there are some there's some work out there in terms of weak supervision where you basically give it sort of image level or point level um training data and uh uh you know I can give some reasonable results in terms of uh training without as much effort in terms of trying to build up those um uh uh labeled uh training data sets.
Um and while this is sort of early work for us in terms of you know um taking it from proof of concept to trying to scale it. Um so in principle um the way you would treat these sort of pipelines is very much um how you would do any sort of um image um computer vision sort of um workflows within the likes of data brick. So um so we can leverage things like ML flow and uh mosaic streaming etc. uh to build up those um training and um uh prediction uh pipelines.
Um I think the key differences are I suppose one how we bring the data in. So the fact that you're trying to read that raw data initially um from those sort of geospatial uh raster formats um you're then trying to keep the metadata intact as well. So the fact that you know these you know these aren't just pixel coordinates they're actually word world coordinates that we have to keep track of um and then once we get the result out the other end it's about bringing that doing the post-processing we need to bring that back into a result that we can uh use.
So for instance you know if I get a um an image level segmentation which sort of represents okay well here's the sort of uh vegetation zone classification for every pixel in my image. Um, the way I want to I don't want to consume that just as a as an image. I want to be able to sort of identify the bounding boxes of of different regions of vegetation. So, usually there's a bit pro pre-processing or sorry, post-processing that we need to do uh there as well.
Um, but other than that, um, a lot of what applies to you building efficient scalable computer vision uh, pipelines uh, applies to geospatial data as well. Um but it's not without its challenges. So uh key things that we have to think about here are things like uh seasonal and regional variation. So you know if I look at a forest or if I look at salt marsh or whatever um the way the vegetation looks in December, it could be very different from how it looks in uh June when maybe some of those plants are in full foliage or or they're uh in bloom etc.
Um so a lot of sort of intraclass variation I suppose based on the time of year um and the region uh that you're looking at. Um we've got sort of tile boundary effects to take into account. So you don't want your model um coming up with a a prediction that you know is says one thing for one tile at the boundary but then when you're trying to combine all of those different tiles together you want to sort of have a consistent result.
Um change detection is is an interesting one as well because you know quite often you know the machine will get you so far but there's still an element of you know you need to have the human in the loop here for re review and quality control. Uh so trying to sort of identify you know has a change occurred you know what what tiles what areas do we actually have to get a human in the loop here to review um uh that output um is something that we have to look at as well.
Um so that sort of takes to that final point around you know the accurate ground truthing uh is really important uh really important to work very closely with the experts and the scientists um in these areas. Uh so for example again if I go back to salt marsh um the way this is done typically is it's literally somebody goes out with their their boots on into the marsh puts down a set square and has to you know bas literally count like the number of different species or what species are present at that uh location.
Um so you know that's really important for accuracy. So where the I think the machine learning can help here is it can sort of help identify those overall boundaries quite often but you still need to combine that with the ground truthing for many of these uh use cases um for an accurate result. All right. So in terms of key takeaways, so essentially large scale geospatial analysis is essential for addressing some of these sort of key problems that we face as a society today in terms of feeding ourselves without um uh wrecking the planet um in in the process.
Um integrating data bricks with those sort of existing geospatial technologies um allows us to do this sort of scalable analysis. So again, it's not about replacing our geospatial tool set. It's about uh complementing that when it comes to problems that uh require more scale um uh and more complex analysis. Uh and key enablers there are the sort of existing ecosystem of sort of spark libraries and uh other open-source um tools that we have in the toolbox.
I should say they're commercial offerings as well. So the likes of Ezri for instance do provide uh commercial libraries that can run directly on tops of data on top of data bricks um and there other examples of that as well. Um and uh yeah and again what we're saying there in terms of sort of computer vision which is an interesting area of growth uh for us in in this area. Okay so that is me. Um, as always, we've got the usual slide in terms of completing your survey.
So, uh, hopefully that was of interest to some of you. Thank you. All right. Well, if there's no more questions, you can come up to the speaker afterward. However, I do want to mention that you will need to vacate the room in two minutes because we can get ready for the next speaker. So, thank you for coming and uh hope to see you again. Okay. Thank you.

Learn more about the Databricks Data and AI platform.