Low-Emission Oil & Gas: Engineering the Balance Between Clean and Reliable
Summary
- Generative AI automation: NOV revolutionized drilling mud report processing using AI, automating 300 reports daily with near-perfect accuracy and real-time analytics.
- Unity Catalog governance: BP transformed enterprise data strategy by breaking down silos while maintaining robust security and governance using Databricks Data and AI platform.
- Operational excellence: These case studies show how AI and advanced analytics enable cleaner, more efficient energy operations while maintaining market reliability.
Low-Emission Oil & Gas: Engineering the Balance Between Clean and Reliable

Talk by Ali Marzban, Director of Data Science and Analytics, NOV ; Jay Yoon, Manager of Data Science and Applied AI, NOV ; Krishanu Roy, Lead Architect, bp ; Srinivas Chandolu, Technical lead, BP
Join two energy industry leaders as they showcase groundbreaking applications of AI and data solutions in modern oil and gas operations. NOV demonstrates how their Generative AI pipeline revolutionized drilling mud report processing, automating the analysis of 300 reports daily with near-perfect accuracy and real-time analytics capabilities. BP shares how Unity Catalog has transformed their enterprise-wide data strategy, breaking down silos while maintaining robust governance and security. Together, these case studies illustrate how AI and advanced analytics are enabling cleaner, more efficient energy operations while maintaining the reliability demanded by today's market.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
How did NOV use generative AI to improve operations?
NOV built a generative AI pipeline that automates drilling mud report processing. The system analyzes 300 reports daily with near-perfect accuracy and provides real-time analytics capabilities.
What challenges did NOV solve with AI?
Drilling mud reports from different vendors have inconsistent formats, making manual analysis difficult and error-prone. AI automation solved this by standardizing extraction and analysis across all report types.
How does BP use Unity Catalog?
BP leverages Unity Catalog to transform enterprise-wide data strategy, breaking down silos while maintaining robust governance and security across their data ecosystem.
Why are these applications important for energy?
These AI and advanced analytics applications enable cleaner, more efficient energy operations while maintaining the reliability demanded by today's market, balancing sustainability with performance.
Full transcript
All right, good afternoon everybody. This is one of the last sessions. So appreciate that you guys showed up. Um I'm the director of data science and uh analytics with NOV. I believe this the slides for it. Uh Jay, you want to introduce yourself, please? Hi, I'm a data um manager of data science and applied AI with NOV. Thank you. All right. Um I am going to show you guys a very cool generative AI um case study that we did with NOV.
The work that we are talking about has been presented at OTC. for the people who are not from the energy sector. OTC is offshore technology conference and believe it or not the number of people at OTC is more than the people that we have here. It's about like 30,000 compared to I believe 22,000 that joined um AI summit today. So there is much more people a little bit bigger but not as exciting as this one. This is much more exciting. uh but the work that we have has been presented at OTC and there is a paper out if you guys are interested I can send it out to you guys later or reach out or download it online it should be available um the work that I'm going to talk about is about a painoint that we had I was on a panel with a couple of days ago and one of the things that I um mentioned during that panel was that we reached out to reach out to customers and ask about their painoint first before providing an a solution and this is one of the pain points that they have.
Um and we have internally as well. Uh let me give you a quick quiz. Um in this one uh I am show I will show you a page. Uh this is a typical mod report that we're getting from um our other um clients that is coming toward us and it is pretty hectic to find some of the informations in these mod reports that are coming on daily basis. Uh for example in this one if you can find the water volume percentage um that will be great.
I will just give you like 30 seconds or so to try to figure out the water uh volume percentage in this mod report that we are getting from uh one of our vendors. You see it's difficult I told you. So um this is something that we have multiple analysts that are dealing with this on daily basis and each vendor or customer is providing a different type of mod report with a different format on daily basis. It is very difficult to figure out where is every um piece of um information over there and they get that information and try to manually put it inside um the documentations or the PowerBI system that we have and they're repeating this process for more than 60 different variables and there is like 200 300 mod reports coming Again for each analyst it takes like at least an hour to do this like get the right information put it in the right format and as I mentioned all of them are not like each other so the challenges that we had here in the mod reports were first of all the templates are inconsistent they are not look like each other they are different each one of the vendors or the clients is sending a different type of mod report to us and there is a high workload and it is very very time consuming for our analysts to do this on daily basis and they will get bored as well.
Um it has a lot of error as well. If you put some kind of water percentage inside there is a lot of error that is associated with putting these things manually into the system and also we don't have a centralized location to just put them um in our system over there. Mud reports are coming through emails they put them in and then there is nowhere that they are saving it. So the goal for this project was to improve the turnaround for the customers and also make it more accurate and efficient for NOV equipment and also keep NOV as the number one um solid control provider in the market as well.
Now I will leave it to who is much smarter than me to talk about the architecture and the things associated with the smart report project. Okay, thank you. So we're going to briefly overview of the uh current processes that we are dealing with and Oh, I think this is yours. Oh, sorry. So the process before was like this. Uh the MUD reports, we have hundreds of reports per day that are coming from multiple vendors and then they receive the email.
They receive the reports in the email and an engineer or the analyst that we have get the PDF. They will review it and then they will put it inside the template or preconfigured analysis that we have and uh that document is generated. Uh then they may or may not put it in the team shared storage and then out of those reports or that they manually put in there will be a PowerBI that will be shown over there. Um so as you can see it is very very manual work going on here.
It's not automated at all. So this is the process that we are using it right now after this case um has been delivered to us to find a solution using Gen AI. Um we the PDF from the emails will auto through an automated process go to through Genai and then um it automatically deliver the data that we have from the PDF read it and provide it to that manual process that we mentioned automatically just fill it in and also it has a centralized location a cloud storage to put it inside of the our system and it automatically update the PowerBI dashboard. port that we have uh for it the BI dashboard that we have for it.
Um so and these are some of the terminologies that we are uh using as well uh for example for um our system. This is one of the phases that we have done. We are still improving on uh this project as well. So I believe this is yours now. Thank you again. Okay. So I'm going to briefly um overview of the current uh implementation on uh AWS and data bricks together. So when mode report provider are sending emails to dedicated email server or an account then we pick up the mode report from the attachment and it is sent to the designated like um S3 bucket and datab bricks has um access to that S3 bucket and we're picking up and starting to process those mails uh on a event basis.
LLM is picking those uh mod reports and then LLM to extract formulated information and then they're kept in the mode report table in Delta Lake and also we have another route to uh trigger this uh event as well. Individual users receive the email and then they dump all those um mod reports sent to uh our dedicate dedicated email addresses and then it also starts the processing the mod report and also we set up some kind of a additional manual validation um way as well using streamlid.
So mod report, automation, user interface is up and then after like individual um uh employees are just checking their uh employment status and then accessibility to the application and then they can dump those one and then validate visually and then they are uh sent to Delta Lake and then uh all the like mod report and those things are on demand basis accessible from uh PowerBI and datab bricks um to the uh mod report table.
Okay. Okay. And um this is also like uh we added um agentic system. So each of the functionality we are working interactively with the agentic large landed model uh agentic system. So like when output template uh is came to prompt and then uh we send it to multi- aent system and then pre-trained LLM is processing those one and we push it to uh extracted mode report data in uh delta table. This is the processes and yeah this is a little bit more um details about like how we are working with those one some of the OCR some of the like um table managing s um libraries how those are like working together uh showing in this diagram okay uh I'm going to move on to the next page and in order to address the AI hallucination maybe even though like you will figure it out like every different try no matter how well you uh define the prompt sometimes it makes a mistakes in every table sometimes like hey this one supposed to be outer diameter sometimes it is filled in in the inner diameter in order to address those like issues we use some kind of a majority voting system inside of it running in parallel to get uh to reduce this kind errors on the AI hallucination.
This one is available because this is not chatbot. This is just the automation. So we're just running this one parallel in the back and then we're getting more accuracy accuracy achieved throughout the system. Okay. And also uh some of the um overview of the how we utilize and analyze this dashboard for client. So when data are extracted from the mod report those are extracted like this is the just the intermediate form but anyway these are extracted in this way and then kept in delta lake um this is the one of the user interface that we are uh using just a drag and drop file and then like um there's some like configurations you want to do like single run multi-run to reduce the hallucination and what model you want to use After that like these are all saved in the datab bricks as a delta table and uh PowerBI has access to this uh delta table to display in our uh client's on demand dashboard and this is the example how we use PowerBI previously and uh surprisingly um there's like over 40 different like um solid control companies available in the oil and gas field um none of the companies are having this kind of capability even though like from the like AI or data analytics people this is very simple but none of these are available in the market.
So um for example here this is just a historical uh visualization of the how mod reports are being used for the bringing the insight to the customer. For example, here um the from the top dashboard, the blue shows the water volume and uh the yellow shows the solid volume and then orange shows the uh oil volume. You can see like throughout the time what is the like mud composition changes over time. So from the beginning you're using more like a water-based mod uh especially using like vertical sections or some more easy uh well plan and then as time goes by as they move to uh directional drilling or more complicated like well planned and then it slightly or it changes to like different mod composition to manage the u model properties and also in the next graph over here this is also a good KPI that like mod engineers supposed to watch on a daily basis.
But in order to get these information, they are just looking up mod report everything one by one individually and getting this kind of idea. But we can show this one on a one place for every like clients that having access to this one. So this one is showing like how much of the mod that you are losing on a daily like uh mod activities or mod operations and stuff. Okay. And also this one shows that um how mod density changes over like depths mod density visc viscosities and those kind of mod property chemical properties changes over operation and also the next one is showing uh how much of the volume they are drilling over well operations and stuff.
So uh these are the how we use bringing insights to the customer in the dashboard. Okay. And next one two. Yes, it's working. All right. Um, so as you can see on this slide, we have uh about six vendors that are coming to NOV for this solution, this type of solution that we are offering. And there is 80 plus um documents on daily basis uh that is going through this system and uh we have tested it multiple times and we have 99.
9% accuracy which is much higher than whatever that analyst was doing it before that. uh but what I'm trying to show you in the next slide is um a lot of people in different industry and energy industry in a specific are talking about AI and Gen AI and use cases of it and how much uh money it is going to generate for the energy industry. um it is very difficult to calculate that the ROI that you are getting out of an AI solution that you're providing for the company if you are not in tech for example you are in manufacturing or energy is very very difficult to put a dollar sign on it you can have a chatbot to help coding you can have I don't know an internal rack system that is generating some uh useful information for you through chat bots but um it is very difficult to say okay this genai solution that we are offering is providing this amount of money for the company.
Um this project though was one of the uh projects that you can have actual numbers for it. Um so this is the cost effective and scalable. You can see some numbers here. We have about 100 reports per day that is going through the system. Before that we had two or three analysts assigned to these to analyze each one of them. As I mentioned each one of these reports was taking more than an hour to put the data inside of it.
Um some of the documents that we are getting are even a scanned version of the PDF. They may not have the good quality and to be honest I have seen some of the PDFs you cannot differentiate sometimes between a six or an eight in the scanned version. It's very difficult but um the AI system is much is doing a much better job in identifying and differentiating between the numbers and put the right number order. it is costing about 10 cents per report.
So um it is very very cost effective. Just imagine that an hour um time of anal analyst at minimum is definitely more than 10 cents. So this is something that you can show for an ROI or a cost save saving basis for this particular project and it is available 247 as well. I mean um a lot of people that we have uh may not be available 24/7. You'll receive an email from a client on a Friday night. Nobody is there to just put the numbers in.
Uh you need to wait till Monday till somebody comes in and put the numbers. This is automated. This is um getting all all of the data inside of it. Um this is not the only Genai project that we have. We have multiple chat bots and stuff as well. Um but uh and we have something similar to this for the um quotation as well which we streamlined it. We made it really really um streamlined and very fast to generate quotations and stuff and that one has some good ROIs on it as well.
Um we do not have time for the questions and answer unfortunately our colleagues at BP have another presentation after us. So if you guys want to reach out or ask any questions, please just stop by um after the session and we'll be available outside to talk to you. Thank you. Appreciate it. Thank you. about our data management journey, how it evolved, the challenges we faced and how unity catalog helped us build a more governed and uh scalable foundation for data in BP.
So regardless of the industry we are in the challenges are the same. The story is the same right we are generating massive volumes of data but collecting the data is the easy part. converting it into something useful, something trustworthy, something governed, something scalable is the challenge, right? And BP didn't happen overnight. It took years of iteration, self-arning and sometimes we had to reevaluate our complete journey.
All right. So, where was I? Yeah. Okay. So, we didn't so at BP this journey this journey didn't happen overnight. We have years of iteration, self-arning and at times we have to reevaluate our directions entirely and this journey still continues. So we're going to walk you through this journey not as a blueprint but as a real world example of how a large scale data architecture can evolve and how data bricks uni catalog can helped us along the way to govern and actually uh address many of the challenges along the way.
Right. I think you guys have to come and press it for me. All right. So, I'm totally take take you through our landscape and our philosophy. At BP, we see data as something that increases in value as is defined from raw facts to information, knowledge and ultimately wisdom and insight and actually it has a big business impact on everything that we have. So as you can see we operate at a very large scale right large scale for us.
I mean 80 billion data streaming data points, two billion streaming tags. At that scale we cannot treat data management as an afterthought. Our philosophy has three uh key areas uh three three items that we look after. One is accessibility. Data should be easy to find and use. Reusability. Data should be sharable across use cases and scalability with governance. For us, it is important that innovation isn't slowed down by uh risk or complexity.
So this balance between innovation and control and empowerment is something that we want to build towards. Next, please. Yep. No. What happened? I'm going to put the whole thing back. There was no automation on this has automation. All right. So, I'll start when the when I I finish it'll probably reach the end. Right. So, our journey started at uh 2014. Uh that's when we set up our first on-prim upstream data lake. At the time it was mostly serve our data to external vendors, a Hadoop implementation, nothing too complicated, right?
2015 2016 we started building our streaming capabilities. 2016 we laid the foundation to shift to public cloud and we migrated our upstream data lake to Azure by 2017. After 2019 we started making key changes to our data architecture. We started exploring architecture models like data mesh data fabric and at the same time we introduced data bricks as our compute layer. Since by the time it was a proven and reliable option for our scale and needs u as our at this point our data environment was maturing and it was clear that our governance framework wasn't keeping pace.
So we attempted to address this by building our own entitlement and access management framework. However, that approach started showing limitations particularly when new versions of Spark was been released and as our data usage expanded across uh different teams. So by this time our data volume was increasing rapidly so were uh the number of users and use cases and the architecture we had in place wasn't able to support the scale efficiently.
Right? So in 2022, we reassessed and introduced the data hub approach to bring in more consistency and structure. While it was able to address some of the governance issues, it wasn't fully able to resolve the fragmentation. Teams still operated in silos and the experience around data discoverability and usability wasn't consistent across the different hubs. So 2024 it was clear that we needed a more cohesive solution one that provided better governance uh supported collaborations between different uh teams better discoverability.
So that's when we we started to move towards a unified data platform strategy with Unity catalog playing a very central a pivotal role in that strategy. So, I guess I I can this just move like this? Okay. All right. So, as I said, uh our journey was filled with challenges. Okay. Despite strong investment over the past decade, we had no shared platform roadmap. Our governance policies were fragmented. Our KPIs weren't aligned.
And data was being managed differently across different teams. uh uh we were we were we had data bricks for a long time by this time and our our workspaces were proliferating and as I said our governance was fragmented uh there was no central view into usage access or access or value uh audit uh audit trailing audit uh auditing was incomplete and reusability was uh minimal because discoverability was low so in short our platform wasn't resilient enough to uh support the speed and scale at which business was evolving.
We got to go to the next slide. Yeah. All right. So, database unity catalog helped us address many of these challenges head on. We were able to consolidate 200 plus workspaces under single governance layer and that enabled us to have a central access management. uh it provided us with uh capturing data lineage across our data products, better auditability and then helped significantly with our cost cloud cost optimization.
It standardized the user experience between across different users uh make it made it easier for data to be discovered, accessed and shared. And we worked very closely with the data bricks engineering folks to execute this migration efficiently. And this collaboration allowed us to align the governance framework natively with the platform capabilities rather than layering it on as an afterthought. So now we have a platform that is built to support uh federated data models across different systems while providing central visibility and control.
So as we move towards uh unified data platform, we feel we can move faster, work more collaboratively and govern more efficiently without bringing friction into the system. Right? So so the journey isn't still over. I mean there's still a lot of work to be done but we feel we are in a strong position and most importantly we think that end users the right person can find the right data make the right decision at the right time.
So with that I hand it over to who's going to take us through the migration process. how we did that, what we learned and uh what was what made it successful. Sure. Before I go forward into the migration journey, I want to add a couple of more points on this. The features in generally when it comes to the Unity catalog, what drove to this decision also. I want to highlight especially the data sharing capabilities because we have data in different hubs in the past.
But the data sharing within the company or del data sharing to outside the company was a major problem which we had where we are having copies of data being shared. People are doing something and again the digital security was not happy with the way we are handling the things because we actually handle the situation where we copying the data back to a location and then we are manually putting the controls in terms of how to share that data.
So this data sharing capability especially helped us in terms of how efficiently we can share that data. So that like what Krishano was alluding right we were tried we were trying to avoid the duplication of the data all together by sharing the data efficiently in the past because people could not discover the data properly they were just duplicating the data assuming that they don't have the data at all and again when it comes to outside the company also we didn't have efficient mechanism but that is a very important feature which enabled us in this journey and again when it comes to the governance aspect also the discoverability in terms of finding ing the data across the data bricks estate was a major problem in the past.
Now we as as Krishano was talking about we had 250 plus workspaces which we had in the past and again if we didn't have the unity catalog's capability in terms of how to discover the data and technically again it's a major problem which we had in the past which was resolved as a part of this and again the lineage is one more big aspect which we were looking at because we always had a requirement to have a technical lineage of the products what we are building.
If a pipeline builds a pipeline to pipeline right the final asset whatever we are talking about that technical lineage in terms of what took to create that asset that visibility did not exist out of the box and this uh unity catalog gave us that capability which was very very important for us actually and again when it comes to the access management also as a platform team we were actually doing the stuff manually in terms of somebody putting us a request we're going and providing the access but now we got the capabilities where we were able to automatically via API is provided.
We gave us a interface where somebody like a user can come uh to a portal they can request something and the API will be actually granting the access with the right approvals in place. So that is the automated process right now. In the past it was a manual process which took a day or so. Now it is all automatic as long as the approvals are in place. So I want to highlight few points but uh I don't want to go over everything though.
Yeah. So this is the migration journey I want to talk about a little bit more. So in the first step as as everywhere we actually got the executive sponsorship in terms of the sponsorship in terms of the top down because we have like several segments in the company right we have PNO we have business services we have enterprise we have bunch of entities who are supposed to work with us in terms of this journey because all these 250 workspaces are used by different sub entities and entities.
So we got the executive sponsorship in terms of the push for us to go and get the unity catalog migrated. So that executive sponsorship was really really important. So that the executive sponsors who also worked with the datab bricks and got the right support from the databicks also because we were one of the ma first major clients who actually started the unity catalog migration in 2024. At that point of time we did not have the tools and the automation also in place.
As a part of the journey, we worked with data bricks who also worked with us to figure out the automated process through this journey. Right. Uh after the second step we were talking about, we also came up with the FM plan in terms of what are all the tangible benefits and intangible benefits what we going to get out of this unity catalog and we actually went and worked with all the sub entities to say that if we go here this is the value what we get out of it because nobody wants to do some migration just like that right because they have other priorities to take care of right.
So we had that sponsorship and also we worked with all the business entities to get that buy in in terms of why we need unity catalog and its capabilities. And then the third step is the key step where we actually came up with the key design documents in terms of key design decisions in terms of uh what is the meta store design in the sense like how many meta stores what we we are going to have what is the catalog design all that technical design documents we came up with before even we actually engage the users to go and do something because we want to have that store setup and also the final design in terms of how things work and we only worked with the pilot users uh in the sense like we picked up pilot users from different entities and worked with them in terms of uh identifying the use cases after the design is in place.
And then the fourth step we are talking about this is also a critical step where we actually did the scripted automation in terms of doing the upgrades all together. And again one part of it is to update update the scripts and the second part is actually we had we ran into this issue on the second step where we were not able to sync the active directory users efficiently back into the account console because we have hundreds of thousands of users that script that did not even work and we had to go all the way back to the databicks who actually had to bring in some engineering teams and also we also work with the databicks engineering team to get that script in place so that we can move forward with the uh with the migration and we also as Krishanu was talking earlier right we came up with the access management framework where we are able to go and get the access to the users through automation than a manual process what we had in the past that scripting work or the technical migration was done before even the actual migration and then we also came up with like something called a racy matrix which was very very important because in the past when I was in high meta store world things worked differently but when I go back to the unity catalog the things operated completely differently.
So what we did is basically came up with a model in terms of what is the R and R in terms of who does what and also we trained the user saying that in the new world after you go back to the unity catalog this is what is the central team's responsibility this is what your responsibility will this model work because in the past they were able to do everything but in the unity catalog model some of the permissions will be taken away and that should be managed centrally by design.
So we had to train the users on that part and after that we actually did a master plan in terms of how we are going to do the migration. We did the pilot migration for like 20 use cases initially and then we did a mass migration for like 200 plus workspaces across uh entities in 6 months and then this these two are pretty much like we are talking about how did we go and do the weekly reporting etc in terms of how the entities were progressing the unity catalog migration all together and then next so I want to spend few minutes on the tangible benefits in terms of what are the technical benefits, what are the financial benefits, what are the operational improvements and what are the new capabilities what unity catalog gave us the capabilities right one thing I particularly want to call out in this one is actually we had a major issue where we were tied up with a custom entitlement framework which was tied to a version of 9.
1 of databicks runtime so that problem was a BP specific problem because we were using the older runtime which we had a custom framework so now as a part of the upgrade That problem was resolved because we were able to go back to the latest version of the DBR. And this particular one we are also talking about this centralized governance in terms of visibility uh centralized governance really really helped us in terms of how do we handle sensitive data like what what what is called out here and then the in the second one in the financial benefits right we had a job which was actually costing us thousands of dollars a week like a using a overwatch where we are spending the money on that job to figure out how many jobs are what is the cost of each job running across tested.
So now uh databicks unity catalog gives that capabilities out of the box via system tables. So we were able to decommission that job and started saving thousands of uh dollars every week using that uh system tables which are out of the box and we also did the consolidation of several of the works workspaces as a part of this UC migration journey that also significantly uh added to the cost savings for our estate actually and operational improvement.
I already called out about the data access manage management framework which is a very important one where people are really happy with the way they're getting the access to the data because now it's much faster there's no human intervention other than the approvals actually and the last one again you saw in the keynote today morning also right a lot of capabilities whatever are announced today are only because are only possible for BP going forward because we have unity catalog already in place we can use all most of the capabilities tomorrow because we have the Unity catalog already in place and we are pretty much in private preview of most of the features with the databicks already.
Yep. And here uh again key takeaways. Again I want to call out the last particularly because in the past we were only having the data bricks back in Azure but BP decided that we're going to have a data bricks in AWS also as a part of this journey and again we are going to extend the partnership in terms of Palunteer and SAP also where we are going to use the Unity catalog to find the integration between the datab bricks and palenteer and data bricks and SAP also using unity catalog that is one big thing And the first point I want to call out is about the people are always complaining about the accessibility to to the data.
That is actually really really uh easy way to get that data and that is what is the feedback what we're getting from the customers. That's it. Y thank you very thank you very much.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.