Zero-Copy Federated Energy Analytics: ADME + Databricks in Action

Summary

• This video introduces the ADME connector for Azure Databricks, a zero-copy federated integration that lets Databricks compute run directly against OSDU subsurface data in Azure Data Manager for Energy without copying datasets into downstream platforms.
• The connector uses Lakeflow Connect and maintains native Unity Catalog governance and serverless scale, and is on track to become an official Lakeflow Community Connector available to all Databricks customers.
• Andy Corran demonstrates the connector against live ADME data, covering federated table queries, a medallion architecture for OSDU datasets, pipeline monitoring, and a Genie integration for AI-powered recommendations.

Zero-Copy Federated Energy Analytics: ADME + Databricks in Action

Oil and gas companies have standardized on Azure Data Manager for Energy (ADME) as their subsurface system of record, but running analytics and AI on that data has meant copying massive OSDU datasets into downstream platforms, breaking governance and slowing every workflow that follows. In this jointly developed Microsoft and Databricks session, we introduce the ADME connector for Azure Databricks: a zero‑copy, federated path that brings Databricks compute directly to OSDU data, with native Unity Catalog governance and serverless scale. The connector is on track to land as an official Lakeflow Community Connector, giving Databricks customers a governed, supported path into ADME out of the box.
We walk through the architecture, show the connector in action against live ADME, and share how operators across the industry are accelerating subsurface analytics while keeping ADME as the single source of truth.
Talk By: Andy Corran, Principle Product Manager, Microsoft - Azure Databricks ;

Chapters

• 00:00 Introduction: ADME + Databricks zero-copy vision
• 03:10 OSDU standard and energy industry context
• 04:23 ADME as Microsoft's governed OSDU platform
• 08:46 ADME connector architecture and Lakeflow Connect
• 10:57 Demo: ADME workspace and connector setup
• 15:21 Federated table queries: zero-copy in action
• 17:39 Performance benchmarks on OSDU datasets
• 21:58 Medallion architecture for OSDU data
• 28:22 Pipeline monitoring and record counts
• 43:21 AI recommendations and Genie integration
• 47:42 Roadmap: next six months

FAQs

What is Azure Data Manager for Energy (ADME) and how does it relate to OSDU?

Azure Data Manager for Energy (ADME) is Microsoft's managed cloud platform built on the OSDU standard, which is the oil and gas industry's open data framework for subsurface datasets. This video explains that many energy operators have standardized on ADME as their system of record, making it a critical integration target for analytics and AI.

How does the ADME connector for Databricks enable zero-copy analytics?

The ADME connector uses a federated query pattern via Lakeflow Connect, allowing Databricks compute to query OSDU data directly in ADME without extracting and copying it into a separate platform. This video demonstrates federated table queries running against live ADME data while keeping ADME as the single source of truth.

What governance does the ADME and Databricks integration provide?

The connector inherits native Unity Catalog governance, so data lineage, access control, and audit logging apply to federated OSDU queries in the same way they apply to data stored directly in Databricks. This video shows the integration running within the broader Azure ecosystem alongside ADLS, AI services, Delta Sharing, and Power BI.

What is the roadmap for the ADME Lakeflow connector?

According to this video, the ADME connector is on track to land as an official Lakeflow Community Connector, giving all Databricks customers a governed, supported path to ADME data out of the box. The next six months of roadmap are covered, including expanded AI capabilities via Genie integration.

Full transcript

[00:08] All right, thank thanks everyone for uh for joining. Um I'm Andy Kin uh principal product manager in Azure data bricks. And so today what I want to show you guys is uh how data bricks and the Azure data manager for energy platform
[00:21] uh itself referred to as uh admi are kind of coming together to create this zero uh zero copy federated analytics um in energy. So what we have is this uh
[00:33] oil and gas um uh kind of um industry where you have many super majors um uh that operate in this massive subsurface um data estate and there large amounts
[00:47] of data sets. the industry has kind of largely um gravitated towards a standardization what we call OSDU and uh that's a subsurface data that uh also then add which is the Azure data um data
[01:02] platform for uh in Microsoft and Azure um and so what the ask is you know from the industry is you know very simple right is that they're they're keeping um they want to keep the data where it
[01:14] lives right so um this is still a kind a pain point that customers have where they want to process the data at sca at scale but they want to leave the data exactly at its kind of location right
[01:27] and this is in this case it's adme um and so what happens today is you'll see these unnecessary copies of data that that happens and so they're pulling it out of ade and then putting it into other kind of platforms and so the
[01:39] interesting part about this this solution that we've created here is that um you can actually supply you can actually uh position the same kind solution, this federated pattern that we've created and extend that way way
[01:51] way beyond kind of just the energy uh kind of industry itself. And so we can use it in a more broader way where we can pull enterprise data uh states and using lakef connect. Um this is kind of
[02:03] our solution ultimately. And so with that um let's get into a little bit more. Um one of our biggest strengths uh that we have from an Azure data bricks perspective today is that you we're
[02:15] deeply integrated right. So it's already there um through the Azure ecosystem itself. um customers are already using it alongside other kind of services that we have things like ADLS uh AI services
[02:28] delta sharing and kind of PowerBI for their enterprise type reporting and now we're kind of adding ADME to that list essentially. So this is a pretty good
[02:40] example in general of um how we're actually integrating this um specific uh lakeflow connect at kind of scale uh within the Azure ecosystem itself. So um
[02:53] by kind of uh creating this Azure firstparty uh kind of integration model um we can now natively uh uh connect the Azure data bricks into the adme uh kind of platform.
[03:10] So before I actually kind of jump into uh the the problem statement we're actually solving here is that um we we want to kind of uh just ground ground ourselves a little bit of what energy industry kind of uses typically and uh
[03:23] I'll start with OSDU right so this became this kind of open standard uh kind of centralized subsurface data for uh wells and data uh in the energy industry itself and this is really kind
[03:35] of designed um to help standard standardized data models and then at the same time also kind of avoid that vendor locking and so with that we have admin which is the Microsoft manage
[03:47] implementation of this uh you know platform and this increas increasingly kind of became um this govern system of record as a as a kind of whole um the same time uh most of our kind of super
[03:59] majors themselves um they're already heavily invested in data bricks right so um this is kind of naturally created did this um uh kind of convergence almost
[04:11] where uh we want to put these two platforms together and ultimately kind of lead to the challenge where we're trying to solve for today.
[04:23] So we know the energy industry is using admin right and so we have this kind of governed kind of system of record but the the moment kind of teams you know start running AI or they start running machine learning or dashboards or
[04:35] advanced kind of types of analytics um what that typically ends up causing is they're they're copying that data right they're copying that data out into these kind of downstream uh kind of platforms and with that that then basically starts
[04:49] to break down, right? We start to lose governance and these downstream copies then create separate pipelines. They're uh duplicated data sets. They're different types of security models. And
[05:01] then on top of that, you've now got this uh inconsistent enforcement of of kind of governance in your data itself. And it's all around the same data. And so in other words, what I'm trying to say is that we the challenge we have is that
[05:16] governance essentially stops where the data started. So that's that statement right at the bottom. I'll repeat it. It's the as soon as you start copying that data out, governance has just stopped, right? You you you basically
[05:28] have zero governance. Once somebody starts copying out, you have no idea where that's going to go. and and so how do we actually enable analytics and AI directly on our kind of subsurface data itself
[05:41] and I would say that you know creating these uncontrolled downstreams of uh data copies outside of the platform that's that was one of our kind of biggest challenges to kind of solve here
[05:54] and then this is exactly what we're kind of designing this whole kind of solution for and what we actually did is we kind of co-developed this with data bricks um and created this Nate flow created this
[06:06] na native kind of lakeflow connector um and some of the super majors also kind of assisted with that at the same time but this this actually allowed us to actually create um this Azure data bricks federated managed adme uh
[06:22] platform. So what I want to walk through um is this kind of unified open architecture that that kind of makes this up right. So we have the raw managed energy data
[06:35] itself and we combine that with the existing enterprise data and applications and we're all actually doing this you know completely within Azure data bricks. So you can see at a high level um from
[06:49] an architecture stack what this kind of solution starts to look like and we're we're start at the top here and with that you have um the foundation which is the OSDU subsurface data itself and this
[07:02] data is all stored in Azure ADLS and Gen 2 storage and then we go down a layer now we can see that the adme provides that kind of govern platform and the APIs that can then expose that data
[07:16] estate itself and that is more secure. It's now consistent. And then on top of that, then we have kind of the the fourth layer down that operates directly against those APIs that are exposed
[07:28] through adme. And so when we start to kind of put this all together inside data bricks, we're using open DT pipelines together with Unity catalog. On top of that, we're
[07:41] actually seeing that we're able to figure out how we can actually scale this out and then you can start using the analytics. Then you can start using the AI features and then you're not having these uncontrolled downstream copies of data at the same time. And the
[07:55] result that we've actually seen from this is that this this is now kind of this fully open native kind of connector that we've created that um essentially enables the the business data to actually be layered on top of that. Um,
[08:09] and this is specifically energy data that we're kind of merging with enterprise data. And we've created this connector that's a fully uh community open- source connector.
[08:21] And when we start looking at what that connector is now made up of um from a lakeflow connect perspective, this really starts to bring together um the the capability where we now have this
[08:33] fully managed native uh federate federation without customers needing to actually do anything when they start using this connector. What I mean by that is that you you don't actually have to do any custom API plan uh plumbing at
[08:46] all because that's all done within the connector itself. You don't have to create the pipelines. We've already done that piece for you. Right? So, um this now starts to look like this zero copy federation that at the core of the
[08:59] solution, what we're seeing is this this native first Azure um federation model that is kind of built directly into data bricks. And then when we start federating the
[09:11] data, what's our key thing that we want to do is we want we don't want to kind of just ingest all the data. All we're actually doing is querying it. We're we're essentially just taking the incremental changes that come in through adme um and we can literally have
[09:25] millions of records an hour that will come in as part of that. And so when there's pipelines come in, we're actually securing it all of this with with managed identity. So we're not
[09:37] actually having any additional kind of client IDs or secrets or anything like that. And I'll kind of go into that a little bit more when we start, you know, start to look at the demo.
[09:52] All right. So when when uh we kind of looked to see how we can bring all this to life is that um we built two I built two demos out of this and and the first one is you know focus more around the operational uh kind of side of the connector and and kind of seeing how
[10:04] that kind of functions and then the second is you know how we can bring what becomes possible now we actually have that data of ad from adme and then we've got it inside data bricks at this point.
[10:16] So let me kind of just give it a little bit more kind of context right around that. So um the admin center itself I'll go through what uh it takes to actually set up the connector um in data bricks.
[10:28] We'll start looking at what the live pipelines actually look like and then we'll we'll see how the data actually lands in data bricks itself and then utilizing the medallion architecture and then see how this kind of fully managed
[10:41] uh operational um connector works end to end.
[10:57] All right. So, this is my um admin workspace and I kind of want to walk through the setup essentially of what it takes to enable this connector. And the first thing I'll do is go down to data integration.
[11:10] And these are your kind of native data bricks uh connectors that you see you see here. Um and they're m they're fully managed by data bricks, but then you also have partner partner ones as well. But this is a an actual beta um option
[11:24] for the community connectors. So you will actually need to uh enable this through the preview in your actual account. So I just go ahead and do this. So you search for uh community.
[11:40] You see that pop up as our community connectors and we can just go ahead and enable that. go back to Oh, you will need to actually refresh um this as well in order for it
[11:54] to kind of show up. Let's go down to data in inestion again. This is where we're going to actually configure the actual uh connector itself and you'll be able to see how quickly we
[12:07] can do this. So now you see in this section we've got the admin uh the community connectors. I've got my admin connector here listed and so let's go into that.
[12:22] All right. So now um this is the kind of super simple kind of setup that we've created. Create the connection. Uh you give it a connector name. I call mine admin production
[12:37] and then next uh you can give it a base URL. Um so you take your admin instance endpoint and you can get that in the portal. I'm just going to paste that in. Then we got the actual partition ID. Um I'm using the default partition ID for
[12:49] your admin uh data itself. And so we just use open dez and then uh the authentication that you're using in your admin instance. Again all of these are actually in your
[13:01] in the portal that you can see all these kind of um configuration properties. And so the the last piece is the orth model. Um how you want to authenticate. Uh so you can use manage identity, you
[13:14] can use service principles. Um in in this case we wanted the most secure way which is manage identity. So that's what I'm going to use.
[13:27] And with that the last piece we need to put in here is the tenant ID. This is your Azure data bricks tenant ID. Um and again get that from the portal. And that's pretty much it. And and so you can decide to use service principles
[13:40] or um use a client ID and a client secret if you choose or if you want to you can actually use a federated token um for dev purposes. But we're just going to go manage identity uh today because that's the most kind of secure
[13:53] way using secretus. Great. All right. So next we're going to actually set up the actual ingestion capability and same kind of deal. We're going to give it an actual name and I'm
[14:07] going to be super creative and call it add me pipeline and then we'll set up our log extraction uh extraction location. So select a
[14:19] catalog. Um I'm going to use my admin catalog I already set up and then here you can select your actual schema. Again, you get list of all the available schemas, but I've got my admin connect adm that I use.
[14:32] And then lastly, you've got the root path, and this is where you're going to store your actual connector assets. So, I've got a folder called demo, and I'm just going to select that guy.
[14:44] Okay. All right. And so, go off and create that. What this will actually do as part of that, it would go off to GitHub and actually download the
[14:56] actual assets um and put it into my demo folder and then it will then create that pipeline for me automatically.
[15:08] There we go. So, I get a notification. It's now creating the pipeline. Redirects me to uh do the last piece of the configuration here. And so this this part here you you can do full configuration if you choose where you want to create a separate destination uh
[15:21] table um to what's actually an adme or you can actually just take the simple approach where we already know in the connector by default the domains so it's wellbor reservoir rockfill um and so I'm
[15:34] going to go with the simple kind of option here and um basically just update with a whole kind of block select this
[15:49] Okay. So you can see we we know what the source table is. Um and the destination is going to be wellbor. What will happen here is it will go off and create those tables for us. Um once I click create pipeline. So if not exist it will then create that for me.
[16:05] Okay. So what it's doing now is spinning up a cluster um a compute cluster and so it can then start processing this uh this pipeline um and while it's doing that I'm just going to jump to my kind of pipeline view it directly and kind of see what's happening.
[16:18] So we can see it's still starting the cluster here um it's basically waiting for the cluster to start and then it can start executing that pipeline. I'm just gonna see if I can refresh.
[16:34] Okay, so everything's looking kind of healthy. Um, which is good. So, um, I'm going to basically, you can see the the actual source tables from ADME here. Um, the destination, it's already created those
[16:46] tables and the actual, um, streaming should actually be starting already. So, let's just see if we can expand that view out.
[17:00] Okay, so we can see on the left there, it's actually syncing. and starting to uh stream. So, I'm going to just have a look at the catalog and see what's coming in. So, I'm just going to look at my reservoir table. So, that's created that
[17:12] for me. Um I can now go and see some data and see how that's landing in. Okay, great.
[17:26] All right, so we can see data starting to come in. Um I'm going to go back to the pipeline and what I want to show I can just type ad me here. because you can do this in data bricks and it knows where it's at.
[17:39] Okay, so I want to show you guys some performance and the performance is is kind of super important here. Um, and so we can see we're doing millions of records here right now. And so that's consistently hitting like thousands of records a second. Um, and this is now
[17:52] starting to solve that problem that we've got and and processing and when you're federating data. And so with that at you know performance at scale um you know we're now able to kind of run this you know fully managed
[18:05] connector in a very kind of simple way right so we're essentially literally in minutes we were able to kind of set this connector up and get the tables created and start streaming that data in um
[18:17] which is which is uh pretty awesome and um especially for the size m amount of data we're talking out talking about from an energy perspective and um what I wanted to show you guys is uh what the
[18:29] command center looks like that we've created on top of this. Um and this is kind of uh where we've really really kind of layered the the advanced kind of pieces that you get as part of this connector. Um and so we created a data
[18:42] bricks app for that. Um so I'm going to go off and um go to my data bricks app. So this is this is native in data bricks. Um, and so, uh, it's connecting
[18:56] to my DTS, my my notebooks, um, and and pipelines here. All right. So, we can see our data bricks app is up and running and live. And I just go into that.
[19:10] So, what I want to do is kind of just level set here a little bit of what we just saw, right? What we just kind of implemented and then how how that all starts to come to come together. Um and
[19:22] so when we start looking at what what we've got when we created the connector right we basically um you know have this admin um into data bricks right on the left hand side and so this is hosting
[19:35] all that kind of subsurface data uh that we have um and so when we start looking at uh the data across our DDMS um domains and the the connector starts bringing that those configured core um
[19:48] domains into into um data bricks we have you know at the top is kind of our wellbor which is um that's essentially your your your depth location your your
[20:01] facilities um and th those sort of properties in that specific subsurface domain. Um then we have like reservoir. Uh reservoir is your your kind of units, right? And and so uh the units is like a
[20:14] field context and those sort of things. Then rock and fluid uh on top of that which are rock samples and fluid properties uh that make that up. So most importantly this is got to follow that OSDU um standard schema. So it's all
[20:28] consistent across across domains, right? Um and and this this now um essentially enables us to kind of really start digging into the heart of what the ADME connector actually does. Right? So if we
[20:41] just kind of look at um what really drives this end to end um at the top we have Entra which is all our identification um and identity um and and so um orth is is is part of this. We
[20:55] use manage identity. It's completely secretless. Um, and so, um, that's that's using modern kind of, um, rather than just service principles. Um, we're able to kind of pull the data in and
[21:07] these kind of chunked reads and incremental updates. Um, and everything's fully fully config, right? There's there's no code behind behind this, right? All we're actually doing is if we want to add more domains, we can
[21:20] just do that through a YAML config. Um, no coding no coding is required. Um and then um the last piece you know we've got here down at the bottom is the uh the delta writer itself. Um and what
[21:34] this is actually doing is it's essentially taking those OSDU uh JSON uh responses from the actual ad uh API and then it's structuring that into DT.
[21:46] Okay. And then lastly at the bottom we've got checkpoints. And checkpoints track the changes, right? So you know we're we're able to kind of process the data that we already know about. We
[21:58] won't reprocess that. So when I go into the medallion architecture here, um this is this is how the data starts getting structured, right? And we have the bronze bronze
[22:10] layer at the top. That's your raw data of OSD records that are ingested um as metadata, right? And then uh secondly, we got like the silver layer, right? the data gets ingested um you know from
[22:22] bronze and that's cleaned and then it's flattened out um and that's normalizing the data on top of that and then uh lastly then we've got the the gold which is all our business ready type data
[22:35] right and when I say business ready that's kind of where adme uh gets the data in um to data bricks and then we join that with uh enterprise data to then enrich that data on top of so we
[22:47] can do things like analytics and and reporting all that kind of good stuff. And so netn net um out of this is that data bricks is the medallion we're using the medallion architecture pattern. Um ultimately um we're able to kind of take
[23:00] the raw data in. So we have we know what the raw data initial raw data is. We're able to clean that and then we can then follow through this kind of simple scalable kind of pipeline on top of it.
[23:12] All right. So let's look at a pipeline and just kind of see what this looks like in in real time. All right. So here here here's this uh pipeline kind of running and it shows you what what this looks like. Um and we're pulling the
[23:25] data in from admin here and then we're actually uh essentially taking each domain and processing that data and actually processes that in in real time and now we have this fully managed uh
[23:40] ingestion of um the the metadata um directly from that. two seconds on that. Okay. Um and then if we look at what the actual um the
[23:53] bronze layer and the silver layer on top of that is that um that's basically processing that flow um of kind of data through through the actual um the
[24:05] pipeline.
[24:18] All right. So, um once it's kind of coming through through the medallion architecture, we're um we we now need to kind of look at uh the security of that metadata bringing in from Adme. Adme already has that that security settings. So, it has legal tags, has the
[24:30] entitlements, it has your ACL's, and these are all mirrored into into kind of the Unity catalog. We're not we're not um essentially having to set this. we're we're copying exactly what's already set
[24:42] within adme and so it's set it once um and so there's nothing for us to do like I mentioned earlier on we're doing kind of checkpointing um on top of this so we can track those changes we can set those kind of watermarks um with it and then
[24:56] um we're then able to kind of process that new data as it as it kind of lands into uh the the gold layer and so when we start looking at this this this whole kind of live uh
[25:08] continuous pipeline of of kind of turning this raw adme um data it's it's coming in to make it all production and insight ready ultimately. So with that let's go to domains
[25:22] and I want to kind of show you here um this this gives us that holistic kind of view of real time kind of admin connected data that's coming in. I can see that it's doing incremental changes. Um I can see that it's running it's
[25:35] active for each each of those domains. Um and then uh I can see that each each layer right across the the board is is actually you know streaming that actively. Um and every single layer
[25:48] let's go here and then our legal tags same kind of deal right so we now have a full inventory of what our kind of legal tags are um and we can see the the synced uh entitlements and alo all our
[26:01] row level security mappings um that go with that.
[26:15] directly to our ad me data now and um this this basically is is using that uh ingested ingested data and I can ask it questions um you know things like you
[26:27] know what domains have been ingested right so it now knows about that it knows that data and uh I get a pretty fast response back and now I can see the domains so let's ask it about reservoir
[26:40] domain as an example. Tell me about what the actual reservoir domain is.
[26:53] Okay. So again very fast, very responsive and we can actually take this a step further, right? And and so we can actually now from here we can also use um a genie space to do um some of our AI BI, right?
[27:08] and um we can get that more deeper reasoning um behind that. It's not just this kind of basic chat um that that we're kind of seeing. So I'm just going to go ahead and go to my genie space
[27:21] directly in the console and as part of that we should be able to kind of see what um genie spaces I have access to. Right? So there are other spaces in this workspace. Um but I'm
[27:34] only going to get I only see my oil and gas one for analysis. So this this is essentially where you can ask it a little bit more kind of advanced. So you can see here um I'm not
[27:46] using the prompt agent. I'm actually using a real agent that can um essentially do a little bit more kind of deeper analysis against my data. Uh so we can go ahead and uh ask it a
[27:58] question. So, um, I'm just going to paste this in. When was the last successful injection for each domain? I want to see operationally what's actually happening here. And so, I can use my GAN to to operationally see um
[28:10] what's really kind of happening um with my connector. So, while this goes through, okay, so I get a domain level injection summary out
[28:22] of this. Shows me my record counts. I get my timestamps and and with that um I can also hopefully get a visualization down here. And so now I can see how many
[28:34] domains um it it basically uh knows about and how many records it's actually ingested for each of those domains. And so um I get a summary at the bottom and
[28:46] that shows me about 1.8 million records that are streamed in. Okay, let's go back to um the command center. And so we we can
[29:00] essentially use Genie for investigating this, but I can also use J dashboards and create um dashboards against my my kind of data as well. So I've seen overview of you know my data. I can look
[29:12] at my operations and so I can look at my governance and see what's actually going on. Okay, let's flip back. All right. So now now I kind of been
[29:25] able to ask natural questions, but I also want to see visuals. So let me go to my data explorer and I can actually start getting those insights to see what um volumes of data is actually being um ingested here as well. Um at the same
[29:38] time I can look at um any kind of failures that that might happen. So, if I go into my change change feed, I can see exactly, you know, what's happening and if there's any issues, then I can go ahead and dig into that um and trigger
[29:51] any sync actions that might actually be needed. So, I'm going to go into my end to end test. And here, this is where uh things get get kind of pretty interesting. So, um I can do a pull of the data. And so, what that's going to
[30:04] do is go off to add me, pull that pull the records, um a subset of those latest records. And let's say I want to make a change to say I don't know the legal tag. Um I can actually uh say change I
[30:16] want to change this to restricted PII right and um it was incorrect and I need to set it correctly but I need to put it back to the source. I need that to go back to adme and so I can actually do a
[30:28] push of that change back to admi at this this juncture. Um, and then on top of that, so we can see that change. And so then I can do a pull and it also then tells me now here's one record that's
[30:40] just changed. And so I can see that record. And this is starting to give this true kind of birectional integration um that we've actually created here. And so let's say I want to make sure
[30:52] that things are ingesting um correctly and we've actually got usable data out of this. Um, we actually have this validation section. And with that um I can monitor the observability um of my
[31:05] kind of data and my subsurf sub subsurface data streams um and that's now able to kind of see if my jobs are actually running correctly. Um but at the same time I can also manually do
[31:29] console at all. Right. So I can do it for governance on top of that. Um but I'm able to kind of monitor my pipeline state here and and see what's happening in real time. I would say with that then
[31:42] as these these kind of jobs run it's it's important to to kind of see if I can actually do any other custom pipelines on top of that. Um and so if I wanted to do a machine learning uh
[31:54] pipeline for a custom schema, I could actually go and set that and and trigger that at the same time. And we can see our live run that I triggered earlier on actually executed.
[32:08] So with that um you're you're essentially able to um get everything without leaving the console itself or this uh the application rather. Um and so lastly I want to show is that the geospatial awareness. Um, so we can
[32:22] see all the wells, the reservoirs and we can see that at a geographic level, right? And I can basically if I want to kind of just drill into say reservoir, um, I can change that kind of view and
[32:34] then also at the same time um, I can start looking at what the actual last known good data that was queried um, and essentially uh, get the latest uh, latest updates uh, for that um,
[32:48] subsurface context. And so I I think what the key the key takeaway here is that um you across all of this is that we have this now kind of uh fully federated, right? And so um we
[33:02] we're leaving the data at the admin platform, right? And when we leave the data at the admin platform, this is now this kind of scalable kind of zero kind of copy federation that we've
[33:14] implemented here. All right. So let's wrap what this kind of demo really kind of showed is that the adme continues to kind of play this
[33:26] subsurface system of record. Um Azure data bricks becomes the place where you enable its business and the the features directly on top of that. Um and because
[33:38] Unity catalog uh ultimately starts u uh governing all of the downstream uh customer AI and analytics and dashboards um you know we don't have this problem
[33:50] anymore where you have all of this uncontrolled copies of data um and so that's that's the key kind of outcome of this is that you know we've now created this uh you know managed production
[34:04] ready deployable um connector that uh essentially you're able to use things like data brick acid bundles which we did in this case is that we can now deploy that across all environments in this kind of consistent way and what I
[34:18] mean by that is we can have our dev environment uh deployed through a dab and we can also have a test environment and also a production environment and it's consistent it's deployed exactly the same way that you also then have the
[34:30] flexibility with the with the connector is to change how you want to kind of set that up so if it was in dev or test. Uh you can uh essentially uh you know choose to do it manually for your sync or you can have it on schedule or you
[34:43] can what we just showed is you can actually stream it as well. Um and so this actually starts to become super powerful and at the same time like I mentioned you're actually getting ade um mirroring all of those entitlements and
[34:56] those legal tags so you're not setting them um and they're all stored within UC at the same time. And then from there uh you also then have the incremental kind of pieces of this the actual pipeline. So you're only pulling in the changes
[35:08] that that kind of happened at a metadata level. Okay. So with that I want to kind of show you another uh short demo is that how all of this starts to become kind of
[35:20] possible right um now we have the data the admin data uh actually in data bricks right is that uh the first demo focused on the operational kind of side of things and how we govern that
[35:33] federation capability and the operational layers around that. Uh what I want to show here is now that we can take the the the subsurface data and now kind of you know power everything on the
[35:45] business side right and so I want to see uh how I can then do the analytics I want to see how I can do geoysical vis you know visualization or um I want to have dashboard insights on top of that
[35:57] right and then um on top of all of this it then all becomes this kind of unified uh you know foundation um of energy data um and So I do want to note that all of
[36:09] this that we're implementing here is 100% all in data bricks right so we're using and hosting the data bricks apps that we've created here that's all within within data bricks
[36:25] all right so if we jump into kind of this demo is um this is really going to show this subsurface intelligent demo itself and I'm just going to go to our subsurface app.
[36:40] Okay, we can see it's running. So, I'm going to open that. And so, this is kind of that uh single kind of experience, right, that we've got. And um we can essentially have this for engineers or geolog geologists um or
[36:54] uh decision makers in the business. Um and then being able to kind of combine all of this operational data um you know uh and different types of economic data sets on top of that. Um and all this is
[37:06] you know through data bricks right so um I want to kind of just go a little deeper into the main dashboard give you some kind of insights of what this is um we have the the economic data here. So
[37:18] this is a WTI um in in input feed and what this gives is um your your price per barrel of oil. Um and so we pull that in. We can now see kind of
[37:30] operational um uh data coming in and that's layered on um essentially with any kind of issues that we might be have um that come with that. And so this is
[37:42] essentially you know showing us both the technical side and then also the business side um as well. Um and this is this is super key when we want to create this type of visualization. Um and then we can also put issues right
[37:56] we can um put whatever modules we want into into this and different data sets. Um and I think this is this is kind of the the biggest key is this this is actually a fairly modular uh kind of setup. So I've got alerts here as an
[38:09] example. So if they're drilling kind of issues or anomalies I can actually you know surface that here at the same time. Um so this now becomes our kind of single pane of glass um ultimately and
[38:21] you know you're able to kind of see all your subsurface data um you know and perspective. Um I've also got my map here. Um this is my map and well inspection. And so uh
[38:33] what I can actually do is uh understand what my overall field activity is and I can drill down into that and start seeing this um you know more kind of traditional energy interpretation of our
[38:46] kind of workflow and and data. I can inspect the logs. I can look at um you know compare wells here side by side and um I can even uh select different depths
[39:00] that I want to kind of look at. And this makes it very easy to kind of look not necessarily from a a high level monitoring perspective, but I can also then look at a deeper investigation um uh point of view. And and so uh with
[39:13] that you can uh ultimately you know get the operational drilling data uh that you want to see um that an engineer will probably be more um interested in this
[39:25] kind of data point. All right. But I also want to show you what we can see from a a 3D layer. Um so we we have this kind of 3D subsurface viewer. Um and this is super cool, right? So you can actually start um yeah
[39:38] looking at well trajectories. You can uh look at the subsurface data and this is fully interactive, right? So I can zoom in, I can rotate it and I can explore it. I can select different wells um on
[39:50] top of that and then I can also change what I'm looking at. So I can look at temperature or um pressure or parocity and then this gives this natural way of of viewing kind of reservoir for your
[40:02] engineers or geologists. Okay. So with that let's uh combine that right. So now we can actually look at this and you know at um our with our enterprise data right I can pull in
[40:14] economic data here. So this is my price per barrel uh for the last three years, right? And then I can overlay that on top with my capex, right? Here's my business capex that I've got. And now
[40:27] I'm bringing in all this additional kind of data um that essentially kind of shows me, you know, with things over time. I can forecast my summaries out. I can then not just look at well data. Um
[40:39] I'm now able to bring in um business data at the same time. And so we're now able to understand what our costs and what our production, what our performance and plus our operational risk um at the same time. So um the the
[40:55] the cool thing about this is um you're you're able to also get the governance aspect as well, right? And the power of UC is that I've got role-based access and that role based access I can have engineers, right? I can have these
[41:08] operators and they only have access to what they need to have access to. So I can redact the data. Um and so where you have somebody that has maybe sensitive um uh economic data, you don't want to
[41:22] share that out with everybody. You just want executives to see that maybe. And so we're able to keep things secure and it's all governed and it's all through obviously Unity catalog at this point. So
[41:34] um we also we also going to have our kind of implementation of um Genie embedded in here. And this is our knowledge assistant, right? It's not just the standard kind
[41:46] of prompt. We can actually have it do agentic kind of responses at the same time. Um, but this is tied directly to to the rag external business data plus the ade data on top of that. Right? So
[41:58] now now this starts becoming this this kind of multimodel capability where I can tie tie it to different the the g I can tie the genie space to different data sets essentially.
[42:11] And so when we kind of look at this is I can also bring in more advanced kind of capabilities like I did with the um uh the admin center where you're able to kind of bring into a genie space but
[42:24] this is now integrated. You can click through very very quickly and easily down to your genie space and now I can go ahead and um ask any more deeper reasoning kind of questions behind that.
[42:42] All right. Now, my last piece is I want to really kind of show you what um we we wanted to take to the next level here. Um and this is the multi-agent supervisor. is this. This is, you know, the most powerful part um of all of this
[42:55] is that we're now able to use the the data bricks multi-agent supervisor where we can have multiple agents um in that know about special areas from economic to the business data to um you know
[43:08] specific well data um that go with that. And so we have these five separate agents and each evaluate different kind of dimensions based off this. So subsurface operations and your economic and and together this will drive a
[43:21] recommendation. So I'm going to ask you a question here and basically identify um you know which what what what should I actually do um based on uh the data
[43:33] you have uh price per barrel and should I actually go ahead and drill this data and so price per barrel $75 kind of what it is today roughly um you know and so
[43:46] now we're gonna go and validate that
[43:58] which is good and it's basically pulling all that together providing that to the supervisor that then pro you know creates this summary at the end. Um and
[44:10] with that uh it's a it's able to identify yes it's a good it's good to actually based on all the different data points it's good to actually drill at this point. Um, now I want to challenge the process here a little bit and I want
[44:23] to kind of just push this uh supervisor and just make sure it's accurately giving me information. Um, so let's let's ask it, you know, should we so should we actually do an infill in this area based on say canonico? Um, canonico
[44:38] we know had an issue in MPT in one of the previous screens. Um, and uh, so just to kind of um, challenge it a little bit, I'm going to change the price per barrel and I'm going to put
[44:50] that down to like say $30. And with that, we'll kind of see how this kind of responds.
[45:02] Okay, so it's actually basically said we should abandon that. We should not do it. And each agent's responded back correctly. Um, and we can see that uh basically this would put us in 13 negative $13 million uh worth of uh
[45:16] worth of debt if we actually drilled that. So I would say it's probably not a great idea, right, to actually uh do any drilling based off that. But I would say more importantly, what's what's kind of interesting out of this is it it
[45:28] basically takes it, you know, further and now identifies when would it actually be viable to actually go and do that drilling, right? And so it's now providing me guidance on top of that. And there a Gentic um guidance where
[45:41] it's now saying the break even point would actually be $34 a barrel. Um and so this is this is super cool. And so now now I know like if I hit $35, I would be at a break even point. And so it's not just answering it, it's
[45:54] actually giving kind of real guidance at the same time. Um and so this is ultimately, you know, driving those kind of better decisions uh from a business perspective.
[46:07] All right. So I want to just wrap up and kind of show you what we actually have from our final architecture view. We're we're able to bring all this together, right? We um we have all of our uh source data here. We we're using Azure
[46:20] ADLS to bring it in. We're using ADME. We're using the Azure backbone um and as our kind of network and uh to all of these data sets. Um, on top of that, I've got my Lake Flow connector that
[46:33] then uh essentially um puts that puts that data into the metadata into um uh DT and using that Medallion architecture here. Um I'm using Unity catalog on top
[46:46] of that from my governance and then I also have my kind of marketplace catalogs here. Um, and below that we have our uh AI serving endpoints, the multiple um AI endpoints that we
[46:59] created. And then uh we're also using lakebase. We're we're using lakebase to store all of our drilling data um right here. Um and then uh APIs that we're connecting to admin. And then lastly is
[47:13] kind of the applications which is this application and the adme command center um that we're using data bricks apps for. And that's all created in React. Um so this is kind of this unified uh
[47:25] single intelligent uh kind of platform um all kind of built into to data bricks to essentially manage your subsurface um intelligence and and experience uh all all in one platform.
[47:42] Okay. So um I just want to give a quick road map and then we'll wrap it up. Um I know we're slightly over time but um you know what I've showed today is is kind of direct API model uh that's in in incremental kind of extraction. Um it's doing cursor uh pageionation as part of
[47:55] that. Uh we've got checkpoints also at the same time using manage identity so we're super secure. Um and we're able to kind of scale um against the admis themselves. Um and and so the next
[48:08] evolution of this um is that the adme team have uh created this uh analytics consumption zone um capability and it's called ACZ for for short and what this actually can do is it essentially is
[48:22] exporting the admin data and it's putting that um directly into ADAS and what that means is now we have this open delta um paret format directly in storage and so where we are doing a
[48:36] direct query to the API. Now we're just going to go directly to storage which is going to be even more kind of scalable and performant. This takes it to the next level, right? Because now it opens the door for us to essentially start
[48:48] using change data capture. All right, and we can do that through parket federation. And so we're going to get high throughput, right? And that that high throughput is going to give us the streaming access and then the the patterns that go with that. Um but we
[49:00] we'll carry on still supporting direct API. Um that's not going away. we'll keep that within uh the actual framework. So customers can actually choose you know which which federation approach they want to use based on
[49:12] performance based on cost and their operational requirements out of that. Um and so uh more to come in this space and that's what we'll probably see within the next six months. Um I do want to
[49:24] call out um before uh we go is you know this this is kind of it's broader Microsoft and and data teams that kind of made this happen and we co-developed this solution um and this is really kind
[49:36] of a cross company effort uh essentially that we solved a real industry problem here today right and we've enabled AI and analytics against governed subsurface data large amounts of
[49:48] subsurface data uh without creating multiple copies and downstream copies copies that typically happens um you know very consistent consistently in a lot of organizations. So huge thanks to
[50:00] to the team for that and making it happen. Um and thanks everyone for for joining me today.

Learn more about the Databricks Data and AI platform.