Site to Insight: Powering Construction Analytics Through Delta Sharing
Summary
- Secure data sharing architecture: Procore implemented Delta Sharing to securely share large datasets across various BI tools and programming languages, dramatically accelerating time-to-insights for construction industry customers.
- Dynamic filtering at scale: Enable cross-platform view sharing with dynamically filtered subsets of data, allowing each customer to see only their relevant information while sharing a unified data platform.
- Efficient ecosystem: Eliminate the need for data replication by sharing data directly from the source. This fosters a more efficient, collaborative data ecosystem that reduces storage costs and ensures single source of truth.
Site to Insight: Powering Construction Analytics Through Delta Sharing

Talk by Vinodh Thiagarajan, Principal Data Engineer, Procore ; vishnu sreenivasan, Staff Data Engineer, Procore
At Procore, we're transforming the construction industry through innovative data solutions. this video unveils how we've supercharged our analytics offerings using a unified lakehouse architecture and Delta Sharing, delivering game-changing results for our customers and our business and how data professionals can unlock the full potential of their data assets and drive meaningful business outcomes. Key highlights: Learn how we've implemented seamless, secure sharing of large datasets across various BI tools and programming languages, dramatically accelerating time-to-insights for our customers Discover our approach to sharing dynamically filtered subsets of data across our numerous customers with cross-platform view sharing We'll demonstrate how our architecture has eliminated the need for data replication, fostering a more efficient, collaborative data ecosystem
Here’s more to explore:
Open data sharing for data, analytics, and AI: https://www.databricks.com/product/delta-sharing
A new approach to data sharing: https://www.databricks.com/resources/ebook/a-new-approach-to-data-sharing
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
What is Delta Sharing?
Delta Sharing is a secure data sharing protocol that allows organizations to share data across different platforms, BI tools, and programming languages. It eliminates the need for data replication while maintaining security and governance.
How does Procore use Delta Sharing?
Procore shares large construction datasets securely with customers through various BI tools. The platform provides dynamic filtering so each customer sees only their relevant job site and project data while leveraging Procore's unified lakehouse.
What are the business benefits?
Delta Sharing eliminates data replication costs, accelerates time-to-insights for customers, and enables Procore to build efficient analytics offerings. Customers can access data in their preferred tools without managing separate copies.
How does dynamic filtering work?
Dynamic filtering allows customers to see only their own data through views that apply filters at query time. This enables secure, multi-tenant sharing where different customers see appropriate subsets of data without creating separate datasets.
Full transcript
Uh this scene. I want to paint a scene for you. It is uh 6:00 a. m. in uh in the morning, a Monday morning. A Manhattan size skyscraper. You're standing in front of it. The cranes are swinging the steel up in the air and hard hats, people walking around just like the babies out scene, you know. So this scene pretty much u sums up the talk what we want to present today. Powering construction analytics through data sharing.
You know uh an architect in the very same building can pull up the tablet and look at the job metrics what's what's going on in this construction site as well as an owner or an executive who's like thousands of miles away gets the same output. So when we are here to show how delta sharing turns this scattered chaotic job site data into a single trusted language where architects or executives or owner can rely on. So when data flows this securely and freely this doesn't look like a hard labor anymore.
So but let's let's go how we make this leap from a site to the insight. We are both data engineers at Procore. So the this is the session. So introduction like who we are, what do we do, what is our goal, uh what do we do in construction and then we just move on to our to explain the commitment what we have made to construction intelligence and then getting into the actual core of the presentation delta sharing. What is data sharing?
Why do you have to care about it? what was the world before data sharing and what it has come into and then Vishnu steps in and we'll explain how we implemented that within Procore and then we'll go on to a place where what what you have to keep in mind when designing systems around this data sharing and then then we'll get into the customer success story. Usually uh this is a prominent slide in any presentation but we have our customer itself in the room.
Um so he I I think he'll come up and share a story here. So who are we? Procore is a a leading global provider of the construction management software. It's a cloud-based uh platform that connects uh project stakeholders like architects, general contractors, owners, speciality contractors on a single platform. Our vision is to improve the lives of everyone in construction and our mission is to connect everyone construction on a global platform.
As I mentioned before, our commitment to construction intelligence like how do we do this? So it is about about empowering the industry to capture, manage and learn from the data and we do this uh by having a comprehensive data platform that captures this data across all project phases you know and creating this rich foundation and then following that we manage it. We provide the tools. When I say tools, it could be anything with respect to construction like project management, workforce, safety, security.
And we give this to the users who can use this 360 reporting one of our offerings to create holistic views. And then we have this procore analytics is one more offering where they can step in and get actionable insights. And we don't stop there. Now we have this data. We give the data as aggregates to the users. But then we have to learn from this and we are leveraging AI. So our insights provide frictionless contextual information within our application.
It'll help you make datadriven decisions. And then we have this co-pilot, our own AI powered co-pilot which makes proc simpler and more intuitive. You can ask questions in natural languages language and it it'll bring out the citations and documents from your own artifacts whatever you uploaded. And then finally we have the agents and automation capabilities. It'll just automate the redundant tasks and workflows which will make your life easy.
It's like having a experienced construction supervisor with you which will go ahead and do the things for you. So by leveraging this comprehensive construction data sets we help stakeholders transform data into actionable intelligence uh to tackle industry challenges. But what is data sharing? How many of you are using delta sharing here in their daily work? Okay, that put a dent in my confidence that so many are using it.
But yeah, I mean I'm just joking. We have pushed delta sharing to its limits. We have databicks team here to confirm that. I think we are the largest user of delta share here. Uh pushing it to its limits. We have asked always for kota increases. Can you increase this? Can you increase that? Push it to the limit. So here how many of you have are consuming data? Are you are you buying data from someone you you consume or are you selling this data to someone producing data for someone to consume it?
Again the problem is the first thing you'll notice is you get a JSON or a CSV or a park from someone and they will promise a schema and then it it'll be a number or a long or a big but there will be a string in there you know and you you will you'll have this data dep mismatches and this interoperability you know the problem this maze what you see you have to do HTTP FTP whatever TP out there you have to build your own consumption layer to get the data out of that and especially when it comes to unstructured data like documents, images, files, you have this challenges already present right there and this data silos.
So whoever is giving you the data are you relying on an external customer or whoever you're giving the data to you're creating silos. You're you're creating copies of this data continuously and when you when when the data is constantly moving like this first of all it is stale it con cuts down innovation and then it's disparate and when you you open up you welcome security risks and governance headaches right it's it's the innovation slows down so these are all the reasons what we are presenting here that why you should go for data sharing like you know so at a delta share is like it's a secure data share sharing protocol that enables organization to share data across different platform and environments in agnostic way in seamless and efficient and governed manner.
So compared to the previous image, what you saw here when when what you see here is a simple Delta Lake entity and Delta server sits in front of it and the clients they they just feel like they are just uh interacting with a local data set and still sitting on park a S3 or any any block storage but the it it's neat if you if you notice it. So there is no more data silos and uh it feels like local and integrated engineers who are just trying to adopt this will adopt it pretty easily.
Super efficient. You share the data in place. So you don't have copies. You don't have you don't have to ship it somewhere into another using a protocol you know. So it's a open protocol for data. And finally uh you are in control. You have a central centralized access management which means that you have precise control. you can cut off the connections, you can efficiently effectively hold the keys to your data. So you can it's it's secure.
So from construction standpoint uh by using the data sharing these are the advantages what we feel that we have to share with the world like it's like a USBC you know like now every entity is having that so it's a master adopter for all project systems. It's it's cost effective that you're not copying data GBs TBs of data and sharing it again and again to your consumers just keep it one in one one place they come read you control the role level permissions even near real time based upon how you design a system it's near real time and you can share any type so it is not just tables it could be unstructured data it could be images could be volume sometimes even AI models and it's governed and secure you always know who's accessing said said before and it's open standard like I I don't think we have ever seen an open standard established like last two decades fail like open API spec graphql and all these open standards they are they have just built for a long haul data sharing is such a open standard and ultimately data sharing transforms silo data into collaborative project resources it's empowering every stakeholder I will now just hand it.
He's the one who implemented the entire system in our company. Let us go and explain. Thank you. Can you hear me? Good afternoon everyone. U I will dive into the practical ways we are leveraging data sharing in uh construction a domain that generates massive amount of data on a daily basis and uh then we will take a look at high level reference system architecture which illustrates how data sharing integrates within our broader procore platform and finally we'll go through some uh key things to consider while implementing data sharing and then a customer success story that really brings the impact of this work to So uh let's kick things off with a challenge.
I think most of us are familiar with data silos. In uh construction, valuable data often scattered across multiple platforms, regions and teams making it hard to generate insights out of it. For example, if the data collected from the field is not connected with the financial system, that disconnect can hide the financial risks. Our solution built on top of datab bricks lakehouse and data sharing aims to break down these silos.
The goal to elevate the data maturity across the construction industry by creating a connected insight ready environment. So let's start with structure sharing. So here we are talking about clean organized tabular data. The kind of data that drives day-to-day operations in construction. So we are sharing these structured data at various granular levels with our customers like schemas, views, tables and even at specific partition levels.
That means our customers have uh super fine grained control over what gets shared for uh for example a general contractor may want to share the project progress with the owner of the building but they want to uh keep their own financials private or they may want to uh share the data with the trade partners which is in their scope of work. So this kind of sharing or access pattern we were able to achieve using data sharing.
So this also means that the data whoever needs the data have access to it instead of being stuck in silos and we we are sharing this data as a entire data set as well as in an incremental approach. When I say incremental approach our tables are enabled with an change data feed property. So for the recipients they can either consume the whole data set or if they if the table is like having millions of record they can consume like what's changed since the large last update instead of consuming the whole data set.
The result it's more efficient uh less resources and faster performance. Moving on to unstructured data. Think about PDFs, uh videos, documents, things like that. So these all are huge part in construction and we are working towards uh integrating them using datab bricks volume sharing capabilities. Uh this way it's it's not just a number in a table we are sharing anymore. We can also attach or share the related files along with the number.
So that will provide more value to it. Let's say for example there is a safety safety incident record in a table. So along with that safety incident table if you are sharing the safety incident report or a videos or images that gives more power to the data and we are also working towards uh sharing ML models for things like u predictive maintenance or defect detections. So these kind of uh ML model sharing we are working towards it.
So how does this all fit into the tools our customers are already using? Delta sharing supports data bricks to data bricks direct integration and we are taking advantage of that to share within our team or within our organization as well as with the customers who are having data bricks. So they can you know even if they are in a different uh teams or companies or regions they can collaborate without any hassle. When it comes to BI tools like PowerBI or Tableau data sharing works right out of the box.
Um no heavy lifting required. Customers can jump in start working on it. And for advanced use cases where uh the customer or consumers wants to consume the data into their own data warehouse and do further processing, datab bricks provides uh data sharing client libraries and we have uh built uh readyto use Python scripts on top of it which we call custom connectors so that know our recipients can easily integrate that into their existing pipeline regardless of the technical environment.
And our c our custom connectors currently supports syncing data to uh various tools like SQL server, snowflake, uh bitquery and so on. So now we uh talked about uh now we are sharing structured data. What exactly is that structured data? It's curated data or um gold models. So our main goal is to make sure that high quality fresh data is delivered automatically and to pull that off we have built everything using medallion architecture right inside datab bricks.
Here is how it works. So the row events uh from our application database and integration events are landed at bronze layer. The no changes are applied there just a faithful copy of what's coming in. So once data arrived in bronze next it moves to silver layer where we transform it clean it validate it and some turn it in something we call confirmed data set basically usable reliable data set and here all the data quality checks takes place to make sure that data is accurate and consistent and then we hit our gold layer where the data gets aggregated optimized and fine-tuned so that it's analytics ready.
So this all layers bronze, silver and gold are deployed across multiple regions to make sure that we can know we can process locally and scale efficiently. Once data is available in gold, we pull that gold from across multiple regions into our primary region. We call it as analytics region. And from here we share the data with our customers. Now uh let's see how the sharing works. So uh for our customers they can generate the token data sharing token or if they have a datab bricks workspace they can add the identifier through our application procer application.
Once they generate a token or add the identifier behind the scene we have automated jobs which runs on schedule periodically checks any new for new shares or changes in user permissions. So based on each user permission in procore system this job make sure that the users are seeing only the data which they are allowed to see. The entire process is like fully automated. No manual process uh nothing required no manual steps required.
Uh so by building our pipeline using the medallion architecture and u sharing the data via delta sharing we ensure that our customers have always have access to fresh trusted data so that they can make datadriven decision decisions with confidence. So while data sharing provides uh know powerful capabilities there are some uh limitations to be aware of. So first um there are no native drivers for uh manually querying the data for the external recipients.
So they have to either use um you know um APIs or client libraries to access the data. Column level filtering like fine grain access control specific to a column level is not supported in a table or partition levels. So they have to use uh cross view platform sharing. Predicate pushdown capabilities it is supported but it is limited. uh its effectiveness really depends on how it's implemented on both provider and the client side and another imple important limitations around uh is volume sharing on uh datab bricks.
So currently this functionality supports within datab bricks to datab bricks. So for any that means any external recipients they cannot access the shared volume. So now I'll invite our um no customer one of the customer uh Bishop Brown from Wagan Constructions to talk about how this works from uh their end. All right, CTO Construction for three years is a midsize GC. Is it on? It is on. Um, first off, Procore customers.
Anybody you in the audience? Any Procore customers? One. I want to talk to you. I want to talk to you. Great. Okay. Um, so we uh we do a lot of different things with obviously the BI stuff is kind of what the reason that Procore Analytics existed. Um, but we did something I guess kind of novel. Uh we took our Hilty on track data which is our assets, the tools, the drills, the cranes, all that stuff and merged it with u the Procore data that's shared with the analytics and time sheets which is also coming from Procore.
And that's kind of the thing that made the delta sharing what's important is we got to get time sheets from a long range of time and couldn't do that from the API because it would just scraping that API over and over and over again would have been uh too much. So we take the um the data from Hilty, we take the data from Procore and then some internal data that we have on costing uh in data bricks, merge it all together to generate invoices that is how we bill our internal tools to our customers.
Um and that uh that is now automated across the company. Once a month those bills get get uh generated without really any interaction from our staff. And that is a huge change from what it used to be where it was all Excel based. Every project manager had to line out all the tools that they think they have, all the cranes, all that. And then the superintendent, the guy on the field would cross reference it and say, "Yeah, no, we we got rid of that one."
Uh, and they would email all of those to the controller and then she would put the bill together. And if you get an idea like 50 plus people doing that is it's ridiculous and it was very very very inaccurate. So now it's very very exact automated no one has to do anything. So that's how we use delta sharing. It's uh one of the main things but I just wanted to brag about that. Thank you. All right to uh wrap things up.
Let's take a step back and look at the bigger picture of what we have built and why it matters. So we are breaking data silos and creating a unified environment across the construction life cycle. We are securely sharing uh curated structured data with precision and flexibility and we are also working towards uh expanding that to include things like images, videos and ML models so that we can support even more uses use cases down the line.
Our fully automated pipeline ensures high quality fresh data flows seamlessly for internally and externally. So for uh whether through BI tools or our custom connectors or direct integrations, our goal is enable faster datadriven decisions across the construction life system. Thank you.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.