Skip to main content

From Wellhead to Insight: Halliburton’s Real‑Time Data Sharing with Chevron

Summary

  • Halliburton and Chevron demonstrate a cross-organizational data collaboration on the Databricks Data and AI platform, using Delta Sharing to publish curated well completion and treatment datasets directly into Chevron's environment without manual ETL or data movement.
  • The architecture captures high-frequency sensor data at more than 200 variables per second from completion operations, enabling Chevron to detect failure signals within seconds and minutes rather than hours—a capability the companies identify as a competitive necessity.
  • Unity Catalog's model registry enables Halliburton to share trained ML models across organizational boundaries, and recommendations from Halliburton's autonomous well completion system can be shared back to Chevron through the same Delta Sharing infrastructure.

From Wellhead to Insight: Halliburton’s Real‑Time Data Sharing with Chevron

Watch: From Wellhead to Insight: Halliburton’s Real‑Time Data Sharing with Chevron
How can raw well data be made available more quickly and more consistently across organizations, without relying on manual data movement? In this joint session, Halliburton and Chevron showcase a breakthrough in energy data collaboration. Halliburton shares how they can publish curated well completion and treatment datasets directly into Chevron’s environment using Delta Sharing, enabling more timely visibility into well performance, deferment drivers and optimization opportunities. We’ll walk through the technical architecture that allows Halliburton and Chevron to collaborate securely across Azure and AWS clouds without the latency associated with traditional ETL. We will cover the practical lessons learned during implementation — including establishing robust data contracts and security considerations, and discuss the ROI of Delta Sharing. Attendees will leave with a practical, repeatable understanding on how to deliver trusted, live oilfield data to support more efficient analytics and production decisions, with zero‑copy data sharing.
Talk By: Kshitij Dawar, Solution Architect, Chevron ; Shahab Jamali, Chief Data Scientist, Halliburton ;

Chapters

FAQs

What is Delta Sharing and how does it work between Halliburton and Chevron?

Delta Sharing is a zero-copy data sharing protocol that allows Halliburton to publish curated datasets directly into Chevron's Databricks environment without physically moving or duplicating the data. Chevron gains governed, near-real-time access to well completion and treatment records from Halliburton's environment across Azure and AWS clouds, eliminating the traditional ETL duplication at every stage of the pipeline.

Why does near-real-time well data matter for energy operations?

Chevron describes well completion processes that inject thousands of gallons of fluid and chemicals into the subsurface, where small anomalies can compound rapidly into costly failures. The companies found that catching failure signals in seconds and minutes rather than hours is now a competitive advantage rather than a nice-to-have, motivating the investment in real-time data sharing.

What was the previous data sharing architecture and why did it fail?

Before Delta Sharing, the architecture involved data duplication at every stage—from field systems to intermediate stores to the final analytics environment—creating latency, inconsistency, and governance complexity. Each transfer introduced delay and the risk of diverging data states between Halliburton and Chevron, making timely production decisions difficult.

How does Unity Catalog support cross-organizational model sharing in this use case?

Unity Catalog's model registry allows Halliburton to register trained ML models and share them with Chevron's Databricks environment under governed access controls. This means Chevron can apply Halliburton's completion optimization models to its own operational data without Halliburton needing to expose raw training data, and recommendations can flow back through Delta Sharing.

Full transcript

[00:07] Good afternoon everyone. I'm Shatish Dawar from Chevron. I'm a solution architect there. >> And I'm Shahab Jamali from Halliburton. I'm chief data scientist. >> So, today I'm going to walk you through
[00:19] a journey that we are calling from wellhead to insights. Uh essentially, it is a story about how raw operational data gets transformed into actionable
[00:32] insights uh via the magic of modern data architecture. Uh while this is a problem we are we are trying to address an energy sector specific problem, but I think the
[00:44] principles that we'll cover will apply broadly. So, hopefully, you all gain something from this presentation. Okay? Okay, cool. So,
[00:56] uh Okay. So, I'm going to start by quickly touching on a few things about the industry as a a whole. So, Chevron is specifically an energy company. We
[01:10] derive most of the energy via oil and gas operations. Um And while I won't go into the intricacies of exactly what happens, but fundamentally, we are collecting a lot
[01:22] of data. Uh and this all these processes that we conduct require the uh we we we have fundamentally very challenging processes in place.
[01:35] As an example, completions here requires injecting several thousands of gallons of fluid and chemicals into the subsurface. Uh so, there there are obviously things that
[01:47] may go wrong, and we are trying to avoid things that going wrong and catching inefficiencies in in those place. So, uh the goal is essentially to even in this process to generate value from the
[02:00] system. And so, the the pres- in during the presentation, uh you'll see more and more about how specifically we are applying Delta Sharing to this completions process. Okay.
[02:13] So, why does data, specifically near real-time or real-time data, matter in this space? So, it helps us make, obviously, faster decisions. Uh and it helps you reduce inefficiencies and catch any uh
[02:29] uh catch any signs of failures. Uh and for catching these signs of failures, the we need to catch these signals in the seconds and minutes after these events
[02:41] occur and not hours after. Uh and that's been where we have been struggling as an industry, I would say, uh to get the data from our remote operations to our central uh hub as
[02:53] quickly as possible. Okay. So, we have noticed that even like certain small delays have vast compounding uh compounding issues. And we have uh we
[03:06] have also determined that it's no longer uh having access to uh near real-time data is not just a nice-to-have now. It is part of our competitive advantage. So,
[03:18] uh that's why we are trying to build something which gets us as close to that goal as possible. Okay. So, what are we currently or what were we doing pre-Delta Sharing? So, um
[03:33] very broadly, we industry has used, you know, file shares, APIs, and some specific protocols, uh such as with Smel and ETP for those in the energy sector. Uh all of these protocols and techniques
[03:47] have their own difficulties. Uh there's obviously, you know, having inconsistent schemas, uh there's a lot of data handover, multiple points of failure, there's also like custom logic that needs to be
[04:00] developed in this space. But most importantly, if you notice, there are several personas that are involved uh and there are several tools that we used to support each of these processes. And it which leads to multiple points of
[04:13] failure. And so we are trying to get away, trying to find a solution that is uh much more seamless. So what are we looking in a new solution? Whatever solution that may be.
[04:27] So obviously, we want as close to real-time data access as possible. We can't wait hours or days for our to get access to our data. Um so that's that's the first thing. We need access to consistent schemas. Uh
[04:41] one thing we notice is that whenever things uh when when when when whenever pipelines fail, data pipelines fail, we don't know if it's something on our end or if it's on our data provider's end. So uh we need to see the same information.
[04:55] Also, within my own company, we have several teams accessing the same data because we all have access to the similar APIs. And so we're working with different versions of data and different versions of data can lead to different insights gained. And so we
[05:08] don't want that to take place either. We want to work with the same version of data. Uh and finally, we want to do this uh in a secure manner manner and a scalable manner.
[05:20] This is where Delta Sharing comes into play. As you can see, uh it has much fewer personas, much fewer tools involved, much fewer points of failure. Uh
[05:32] it is uh it requires no data copy, no data duplication, uh uh and also it's very well governed. We also don't need to gatekeep data anymore. Uh in our org with Delta
[05:44] Sharing, those who have access to Unity Catalog, uh can also see what tables are available via our data provider. So, it's access should not be limited. We're trying to make the data
[05:56] as as readily available as possible. So, we don't want fragmented data, we don't want inconsistent data, and we want data easy to use. So, uh this helps us get to that space as quickly as possible.
[06:10] I'm going to show very briefly what the pre-Delta sharing architecture looks like. So, uh again, not getting too much into details, what it involves is like from our remote operations, we would traditionally
[06:22] collect information, uh generally with our data providers like Halliburton. Uh once they collect the information, they would generally publish it to a certain data store. Uh we we in the industry
[06:34] it's generally the WITSML store. Uh so, there's a lot of process that goes into their end to make that data available in the WITSML store, and then there's a lot of process for us to extract that information from there.
[06:46] Uh and it's a terribly difficult process. I'm really simplifying the architecture by showing you here showing it here. Each and every single step has several several points of failure. And necessarily requires duplication.
[06:59] So, we duplicate the data when bringing it in. We duplicate the data across every single stage within our own process. And then finally getting the data to our BI users, to our uh um
[07:11] to our data scientists, it's a little difficult and takes a while. So, you can see the post-Delta sharing architecture, where uh we still collect the data from the from our remote
[07:24] operations, uh but now uh Halliburton uh and our data providers get the data into their Unity Catalog, and it's a seamless share into our ecosystem.
[07:36] Uh and within that within our ecosystem, we are then able to have we can have consistent schemas. We don't have to battle. I don't have to fight with folks in Haliburton to figure out who is right and who is wrong. Are we seeing the same
[07:48] data? We know we are seeing the same data and so we can uh so we are we are we are better collaborators now. Uh also being part of having our entire stack within the Databricks ecosystem
[08:00] has made things easier for us as well from a pure machine learning data science BI uh stand standpoint.
[08:12] Before I hand it over to uh to Shahab to talk about how the production side of this works, the data production side of this works, I want to quickly highlight the efficiencies that we are gaining throughout this whole throughout this whole process. So, first of all, we are
[08:26] obviously improve improving our collaboration between our data providers and uh as a Chevron as a whole, but also within our own teams. So, we don't see inconsistent schemas, so we we can speak
[08:38] the same language. Um we are also reducing reducing a lot of overhead. We each hop is an engineer at least or at least a process or point of failure in
[08:50] place. So, we are getting rid of all those points of failure. We have accelerated our BI and machine learning. So, our models, they run on real-time data, give insights in on real or near real-time data. So, no longer do we have
[09:04] to do just descriptive analysis. We are moving to the prescriptive world. Uh and finally, we have faster data availability. So, for our BI analysts and for our uh SMEs, they get to see
[09:18] operation data operational data in real time for efficiency. So, I'm going to hand it over to Shahab to talk about how Haliburton makes this a reality for us. Thank you. >> Thanks, Ashish. Well, so we saw so far how the consumer
[09:32] value is and basically the faster data availability reducing the integration over overhead, improving collaboration, and in general so really it's a reducing the time and
[09:45] effort that it takes to convert data to insight and and analytics. So for us really the challenge is not simply exposing the data. That's a easy
[09:57] thing to do. The challenge is that how do you take physical operation like fracking and convert that to a uh well trusted and governed data products
[10:09] that you can use internally or you can share that with your with your customers. And just why why do we want to do that? I want to give you a little bit more context here. When you look at the
[10:21] fracking and oil business or there is a executive companies like Halliburton. We are executing what basically our customer wants. So they want how they want to treat the well. Basically they come to
[10:34] us with their plan and we execute those plans. But each of us has piece of the puzzle. We know how to execute. We know that how basically well is work. They are they know that how basically reservoir works
[10:46] and they know that how to optimize that. To get maximum value from the asset, these two piece should come together. And that's what we are trying to do with this Delta sharing. It's not just sharing the data. It's a sharing the
[10:59] insight and knowledge about what's happening at the well site. And I'm going to walk you through how basically we achieve that. Uh to get there so I have a very short video. I want to play it and then I will
[11:11] talk about it.
[11:51] >> So what you just seen here is that we convert a physical operation to a digital operating system. And the closest analogy that I find for this one is self-driving car.
[12:04] So in self-driving car, you set the destination that you want to basically go and the car will drive. You are not taking action about changing lane, how to react to the weather, and etc.
[12:17] So AutoFrac uh our product is doing the same thing. So basically Chevron comes with a plan. They will say that how they want to treat the well. And we pass that
[12:29] plan to our system and our system drives. So execute all all those plan automatically. And when I'm saying that executing all all those plan automatically, it doesn't mean that it's just sequence of basically execution. It
[12:43] is aware of what's happening at the surface, how how the health of our equipment is. If there is a problem, it has a self-healing uh mechanism. It knows that how to bring one equipment down, basically use another equipment,
[12:57] or rebalance with the equipment that we have. And we went even one one step further um and if you look at the oil oil and gas industry, so we are not interested in surface, what's happening
[13:10] in those pumps. We are interested in what's happening at the subsurface. So how how are we treating reservoir? And we provide two sensing mechanism for
[13:22] what's happening at the subsurface, which we call it sensory near well. This give you a sensing about what's happening close to the well and the cross well solution which is give you a sense about what's happening further
[13:34] from the well. And those insight or those uh data can be used by operator to decide on how they want to react for the events
[13:46] that's happening at the subsurface. Now, they get that those insight and then they decided how they want to basically react to those events that's happening at the subsurface. Based on their plan, the new plan
[13:58] basically we can pass that to autofrack that I show you, it can basically automatically change the plan and execute that. So, with this one basically we are closing the loop between subsurface and surface.
[14:10] So, we close the loop at the surface by autofrack so we know that how to manage all the equipments. We close the loop between the surface and subsurface. The last piece is the closing the loop between the companies that I'm going to
[14:23] talk about that. But, how did we get here? So, it's really it's not that simple as I show here. And as we see that through this uh summit, the king the data is the king.
[14:35] So, the main thing is the data. And if you go to any Halliburton site frack site, basically all the equipments that we have it's equipped with different sensors. We get lots of data
[14:47] from the state of any equipment that we have uh in the in the site. And just to give you a sense about the size of the data, each pump is
[14:59] generating more than 200 variables with 1 hertz uh frequency. And that's result in about 50 trillion data points per year for only pumps. If
[15:11] you add blender, power generator, uh manifold, all other equipment that we have, this one basically it's 10 times larger. And then I talk about the subsurface sensing, that is using the fiber optic,
[15:25] that those are high frequency about 100 K Hz data that is generating. So, if you look at one a stage, it's it's generating about 100 trillion data points. Uh one a stage is about 3 to 4 hours
[15:39] operation. So, you can see that the the amount of data basically it generate in frac site. And we use all this data every second to optimize operation that we are doing. So, it's a massive work to do.
[15:52] And so, to be able to achieve this one, you need a very good data foundation. Uh this is a schematic of data foundation that we have at Halliburton. We call it Halliburton uh operational
[16:05] backbone, because this is a backbone of everything that we do, analytics, AI, ML. And it's a start from left, so the ingestion part. So, there are two main
[16:17] paths. So, I I simplify that one, but there are main uh there are two main paths. One, event hub like, so this is for high frequency data. So, these are basically Kafka or service bus or event
[16:29] hub that basically ingesting the high frequency data to our backbone. There is a storage account like, which these are the basically a slow changing uh data like basically job information,
[16:42] contracts, SAP information. All of this data basically is land to our backbone. And the first layer basically our landing zone, we try to keep that as close as possible to the day source
[16:55] data. So, that at any time we can go back and replay the whole history. And that's very important for you um if you want basically your backbone be resilient and you be able to add new
[17:09] KPI, you you'll be able to extract new insight based on what you learn through the time, you have to make sure that you capture all the data. You don't want to lose any of data. And then basically we go through them
[17:21] usual medallion structure that you see that in most of the data structure in in data breaks. So through some ETL process we improve the data.
[17:33] When I'm talking about improving data back into AI and everything that we see in the summit. So basically we are making the data contextually richer. So instead of just having the raw data
[17:46] as you go from left to right basically the data is aware about where it's generated. It's belong to what customer, what condition basically that data is generated and so on. And then we get to the point that we we
[18:00] have a data that we call it data product. Those data product is rich on on the metadata and the context and it got preprocessed and curated in a way that is
[18:13] readily available to be used internally by our data scientist or to be shared to with our customers. Um the other two piece that I want to
[18:26] mention here, we have a data governance and security layer to make sure that any steps that we are taking here basically is well governed and secure. And we have a data observer
[18:39] observability layer that basically is give us a very rich information about what's happening to our data, to our pipelines and so on. And this one currently we are basically
[18:51] managing that manually but with the announcement of the zero ops basically this could basically lead we have the foundation to connect that to zero ops and have a automated way to basically
[19:03] diagnose anything that goes wrong with our foundation. So if you look at most of the thing that we are doing the Haliburton operation
[19:15] backbone, it's powered by data data brick products. And it's good to have one place that you can go and basically manage all the pieces that's happening. Uh so quickly, so all of our asset
[19:27] digital assets are sitting in Unity catalog. And that's make sure as Kshitij was talking about that, that make sure that we are governing the schema, we are we are governing the access to any data either in row
[19:41] level or column wise. And then all of our most I should say most of our orchestration also happening in data brick using the lake flow for ingesting
[19:53] other data sources or a spark jobs. Uh Building models, basically we use data brick to build models, ML models, and we use ML flow basically for tracking
[20:06] traces of those model. And all of those model also get registered to Unity catalog so that it will be available internally and externally. And for deployment, we are using model serving by data brick and also the
[20:21] dashboard and app data bricks apps. And finally, the self-service part is because we basically curated this data and make the data product ready and has a rich metadata, basically it can
[20:34] connect we connect that to Genie and you can basically interact with the data by using natural language. So just one one example of how this process is
[20:47] happening. So at the left, you see a some telemetry wave. So this could be a rate pressure fiber data that is coming from the subsurface. Then these are going through
[20:59] the steps that basically, as I said, we are enriching this data. How do we do that? For this example, basically we said that okay, we have this data."
[21:11] What customer basically this job is belong to what customer? So, we add customer information to the data. When was the start and the stop of that stage when we are pumping? So, we add those data uh uh to it. What well was it? So,
[21:24] now we know that the location of that job and so on. So, we make the data basically having more and more context about what's happening compared to just being the raw data and showing that what's the rate, what's the pressure.
[21:36] And then you can do the aggregation level on the on top of this data. So, we we say we generate a view that basically it shows the aggregated level as it at this stage. What is the average pressure? What is the rate average rate
[21:48] and so on. So, this would be a very compact data, but highly basically it has lots of information that can be used by analyst and by uh our customer to uh take decision.
[22:01] And this data now is ready to be shared with our customer. So, how do we share that? Uh she did show that, "Okay, there the traditional way basically it's involved
[22:14] lots of copying the data from one place to another place, generating CSV file, send that through email. When we become very fancy, we use FTP to send the data. Uh but now basically we are at the point
[22:28] that this kind of latency that you get because of all these steps is not acceptable anymore. So, we want basically our customers and even internally our uh analysts, they want to to be able to interact with the data as
[22:42] soon as is basically land our backbone. So, to reduce this latency, we are using this this zero copy architecture, which is coming from Delta Sharing.
[22:54] So, for Delta Sharing um so, basically uh Halliburton gather the data, curate the data, make the data product. Put that in Unity Catalog, catalog. Do the delta sharing.
[23:08] All of these things happening for us, for example, in we are at in Azure cloud. Our customer then we share that that data with our customers. That could be in AWS, any cloud. So then the data,
[23:21] as soon as it's available in Haliburton backbone, it will be available in their Unity Catalog. And they can extract insight through from those data. They can
[23:33] generate new models out of those one. So with this one, there is no duplication, no manual effort. You don't need to care that if the your customer is in AWS, in
[23:45] Google, how you want to handle that. So all of that basically, naturally, it will be taken care of. And I I I wanted to say that this is a simplified version of how you would do
[23:58] that, but really this is very close to how you would do that in Databricks. So let me walk you through this one. So the first line, basically, you generate the object in your Unity
[24:10] Catalog. It's a shared object. This is the object that you want to share with you your customer, your client. Then you decide that what are the digital asset that you want to add to that object. So for example, here I add
[24:24] two uh table to that one. You can add table, you can add model, any digital asset that you have in Unity Catalog, you can add to that shared object. Then you you create the recipient
[24:37] identity. This would be Unity Catalog of, for example, Chevron. So we register that Unity Catalog in our Unity Catalog as a recipient. Last, you grant access to that
[24:49] recipient. And that's it. It's done. Uh so there are some nuances for uh you know you all know that with IT five firewall and so on. But, really what data engineering
[25:02] they need to do is this as steps, and as as simple as this. And then, one thing that you get out of here, we get the grant access to Chevron. And it's be a it's it's going to be
[25:15] available in Chevron Unity Catalog, but they have whole basically governance layer there. They can decide that who in Chevron can access this data. So, we just give the Unity Catalog of Chevron ability to
[25:28] connect to our Unity Catalog for these tables. So, just to close the talk, um we are we have another close look that we build. This is a close up collaboration
[25:40] between us and our uh customer. And I want just to quickly walk through this one. So, basically at the field, we generate the data by executing what has been asked us to do
[25:54] by our customer. And then, we curate that data. We make that a data product. We do the delta sharing. Now, that data is available for our customer. And in addition to data, if we have some
[26:07] recommendation that basically we should change the plan for this well, we can send that that one also through the delta sharing to our customers. Or, our customer can run their model in real time on this data to come up with some
[26:19] proposed uh intervention that they want to do during the job. Then, we get that that in intervention. We get the approval for those intervention. It's pushed to AutoFrac
[26:31] and Zeus IQ. And our self-driving car basically execute those those recommendation. And we generate the new uh data. And this loop is happening basically
[26:43] continuously. Now, with this one, basically we are opening the part of data or part of the space that we weren't exploring before because of the latency that we had. We weren't be able
[26:55] to react to some of the events that's happening that fast, but now we are reacting to those events and we see from this new field response, we can see that okay, the result of those
[27:07] intervention that we are doing what's the result of those one and we can go further refine this uh our our process and going back to our objective, we can really improve our asset and we can
[27:20] reduce the dollar per barrel. With that, I I want to thank you all for joining and I also want to thank Databricks team, Chris and Andrew for all the help that they
[27:32] help and support that you did during this journey. Thank you all. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.