Autonomous Energy Operations: Data and AI at Grid Scale

Summary

• Eversource deployed a medallion architecture on Databricks to process smart meter interval data from millions of customers, establishing trusted data zones and both managed and self-service strategies that reduced billing processing issues by hundreds of hours.
• EDP is pursuing a seven-bet strategy for autonomous energy operations, using Databricks to enable federated team alignment and real-time data from drones and IoT telemetry for production-scale solar asset maintenance across global renewable assets.
• Both utilities demonstrate that a scalable data platform on Databricks is foundational to autonomous grid operations, renewable integration, and data-driven decision-making that improves safety, reliability, and affordability.

Autonomous Energy Operations: Data and AI at Grid Scale

Energy utilities face a critical challenge: massive volumes of operational data from millions of smart meters, remote assets, and grid infrastructure generate thousands of data points per minute, yet most organizations lack the platform and governance to turn this data into real-time action. Eversource and EDP, two major energy providers, transformed operations by building data platforms on Databricks to enable autonomous decision-making, accelerate renewable integration, and drive grid reliability.
Learn how Eversource deployed medallion architecture, established trusted data zones (operational, standardized, product), and implemented both managed and self-service data strategies to reduce billing issues by hundreds of hours. Discover EDP's vision to lead energy transition through data-driven autonomous operations, including their seven-bet strategy for creating business value, federated team alignment, and skills development. See production-scale autonomous solar operations using drones, IoT telemetry, and real-time data to reduce maintenance costs, improve safety, and cut emissions across renewable assets globally.
🤝

Chapters

• 00:00 Introduction and Challenges: Eversource and EDP
• 04:17 Solving with Databricks: Platform and Architecture
• 07:17 Governance and Self-Service Strategy
• 10:31 Data as a Service: Managed and Self-Service
• 14:52 Real Impact: Solving Business Problems with Data
• 19:01 EDP: Vision, Strategy, and Operating System
• 28:24 Autonomous Solar Operations: Challenge to Production Scale

FAQs

How does Eversource use Databricks to process smart meter data?

Eversource collects interval data from millions of smart meters every 5 to 15 minutes and integrates it with their outage management system through a medallion architecture on Databricks. This enables near-real-time visibility into customer outages, transformer loading, and grid usage patterns that were not possible with the previous meter infrastructure.

What is Eversource's trusted data zone strategy?

Eversource established three trusted data zones—operational, standardized, and product—to organize how data moves from raw ingestion to governed, analytics-ready assets. This structure supports both managed data pipelines handled by the central team and self-service capabilities for business teams, and has contributed to reducing billing processing issues by hundreds of hours.

What is EDP's seven-bet strategy for energy transition?

EDP's seven-bet strategy outlines how the company plans to create business value from data-driven autonomous operations, spanning areas such as renewable asset management, grid reliability, and federated team alignment. The strategy is supported by a skills development program ensuring that teams across EDP can contribute to and benefit from the data platform on Databricks.

How does EDP use autonomous operations for solar maintenance?

EDP uses drones and IoT telemetry to collect real-time operational data from solar assets and feeds this into Databricks for analysis. This production-scale approach enables automated detection of maintenance needs, improving safety for field workers, reducing maintenance costs, and cutting emissions associated with unnecessary site visits across their global renewable asset portfolio.

Full transcript

[00:10] Okay. Well, welcome everybody. Um good to see a nice crowd here. Um so I am Brock Wagner and I am here to present with my cohort here Pedro to about the autonomous operations and
[00:26] there's two utility companies presenting. I'm representing Eversource and Pedro's representing EDP. Um So basically I am the um
[00:42] manage strategy and services for Eversource's enterprise data strategy or enterprise data group. Um And also I'm the lead for our AMI analytics
[00:59] program. We just recently launched AMI. I think we're perhaps the one of the last major utilities in the US to launch smart meters. So to give you a little bit of background on who Eversource Energy is,
[01:15] we're a major energy provider distributor in New England. We service gas and electric to the states of Connecticut, Massachusetts and New Hampshire. And like all utilities our number one
[01:32] goal is safety. That's a major focus as well as reliability of course and then doing good. Energy is fairly expensive in New England so affordability is also a major goal focus of ours and we're doing that achieving that through
[01:48] grid modernization and uh uh uh smarter operations empowering our operations teams to make data-driven decisions. Um so I think
[02:04] no surprise to anybody here is the reason why you're at Databricks. Um data is be becoming more and more important for driving utilities. Um the for us um the deployment of smart meters
[02:19] represented multiple orders of magnitude more data to consume and make available to to our internal teams as well as our external customers and you know provides capabilities like alerting uh
[02:38] better visibility. Customers have better visibility into usage. Um and certainly our internal teams require represents the the ability to um transform our business processes. Um
[02:55] but this all requires infrastructure, a platform that can scale to handle this this degree of data. Um the the the use cases that we're supporting kind of like runs the the gamut of our operations.
[03:12] We've got we're pulling in interval data on the order of every 5 to 15 minutes and integrated with our outage management system. So much more better visibility into the outages. So surprising that customers don't realize we have no idea
[03:29] with standard AMR meters whether your power's out or not. So that's a huge huge benefit. Um Uh certainly the ability to support um better decisions with the interval data,
[03:45] loading, and transformer loading is huge to make more proactive decisions on deploying assets. And of course, the regulator uh, is always looking for proof that we're spending the rate payers money prudently. So, that gives
[04:01] us great insight into the effect of of of the decisions we're making. Um, obviously the challenge is turning all this data into um, action. Um, and that's where uh, having a uh, uh, highly scalable platform
[04:17] um, analytics engine, I like to call it, like Data Bricks is is critical. Um, standard uh, on-prem relational databases just cannot handle uh, the volume of data that's coming in. Um, and for us, a big part of our data strategy
[04:35] is um, enabling our businesses to um, get access to the data uh, and perform ad hoc analytics or or even build um, uh, uh, dashboards themselves. So, giving them fast access to trusted data is um,
[04:51] absolutely paramount. Um, and uh, you know, being able to identify issues um, in the field quickly. I've got an example um, later uh, that uh, explains that in an exact use case. Um, and and uh,
[05:08] coming up with a strategy that allows us to uh, reuse the data, uh, reuse the analytics, um, our data products across uh, different um, stakeholders is is really critical to scaling this um, well. Um,
[05:23] give you a little idea of the journey we took. We started with probably as many did um, with many kind of uh, siloed repositories wrapped around specific applications that were being used for specific business processes. That's kind
[05:40] of There's a lot of that still today. So, and and some of it's just the nature of the the Like, you can't move away from uh targeted uh operational analytics. But, um the big um the the big move for us was trying to bring this data into a uh a
[05:58] single place where customers can um create enterprise analytics. That's kind of our our our maturity path into uh an analytics uh platform um maybe about 10 years ago. Um that's where we're moving into Azure. We're an
[06:14] Azure shop. Uh Microsoft shop. Um but, uh but that was uh didn't necessarily scale well. It also didn't necessarily have the governance that we needed. And so, about 5 years ago, we had a change in leadership. That's when we moved on
[06:29] to um Databricks. And um that basically set up the the foundation we have today. Um and that is actually uh what has brought us to where we are today, which is um creating standardized uh reusable
[06:45] data assets and um providing uh a self-service um approach for our customers, which I'll our internal customers, which I'll get into a little bit um in a in a minute. Um Diving down into some of the technology
[07:01] or the technical choices, the architecture. It's a standard medallion architecture, um but I don't necessarily like using the terms bronze, silver, and gold cuz those don't necessarily mean anything to our business. So, we've opted for using terms that actually drive business
[07:17] value. So, uh operational, you notice I kind of color coded um these different zones with bronze, silver, and gold, but uh operational, um that's uh the traditional bronze layer, bringing in, you know, high fidelity high fidelity replicas of the source system.
[07:34] That includes duplicates, any errors in the data, all of that comes in, which is really, really important for building trust because you're in in in giving our stakeholders access to that data so that they can have a dialogue
[07:50] with the source team and just discover issues in the data and get it addressed in the source which is super critical for creating trust all the way down the chain. Next is our standardized layer. That's our semantic layer. Um, we're building
[08:07] out something we call the um the ever source unify data model. It's a following an ontology of I actually like the fact that I heard this a lot at the summit here about context and ontologies um which is super critical because all
[08:23] of the source systems are very vendor specific schemas um which you know takes a while for you to wrap your head around it but the goal of standardized standardized layer is creating reusable data assets that follow naming conventions that the
[08:39] business understands. Um, and and and also that's where you begin to start cleansing the data so anything that would be perceived as a duplicate gets flushed out of the system or flagged for um remediation if it's actually a problem. Uh, and then product zone is where
[08:55] you've got your fit for purpose uh data assets that are powering your um dashboards or um system integrations with other analytics tools and whatnot. So it's kind of pretty standard stuff. Um, and and that actually has set us up.
[09:11] That was actually the foundation for us to move into uh self-service governed self-service um um uh solutions. And and basically again trusted data is the key to all of this and you can't have trusted data unless
[09:27] you trust that the source doesn't have egregious uh errors in it. Um, and uh and that I I'm big fan of saying uh trust but verify. So, I want my customers, my internal customers, to be
[09:43] able to verify that what we're saying is true, and that also builds trust. Um and enables them to actually do the exploration themselves. Now, it's governed, so not everybody gets access to everything because if you don't know
[09:59] what you're doing, you can easily make uh bad assumptions and tell the wrong story. So, governance is a huge huge part of this. Um but like I said earlier, if they have access to the data, they can discover issues and get them resolved much faster. That helps
[10:16] with building trust. Helps with addressing persistent issues that will crop up. And and and so the AMI our AMI analytics program kind of opportunistically drove a lot of this this
[10:31] progress. Um so, taking another step lower and talking about our data as a service approach, we basically look at two classes of service, managed service and self-service.
[10:48] Managed service is traditional IT department owns all of the ingestion, transformations, creating the data products, handing our business stakeholders the the dashboards or the system
[11:03] integrations to analytics tools, and we manage it end to end, which is necessary for some of our business stakeholders. They don't They're not data savvy. They're not necessarily technically capable of building these things themselves.
[11:18] But layered on top is our self-service strategy, and it's essentially taking we call it the analytics workbench, and it's just us branding Databricks and providing the same capabilities that we have internally in our IT
[11:35] uh data engineering teams have for uh ingesting their own data. I call it bring your own data um because there's going to be plenty of data sources that uh different business stakeholders have that we don't necessarily have time to bring on board.
[11:51] So, I definitely want to get out of the way of being a blocker for teams getting their work done. Um they have dedicated compute. Um so, every uh different teams are kind of firewalled from each other um both from a uh data
[12:06] governance perspective cuz they also each have their own catalog. So, they can create their own data product or their own data assets if they want. Um but you don't have to worry about somebody stumbling across and randomly finding someone else's um table that does something that they don't understand and misinterpreting it. So,
[12:22] that's a big part of governance. So, the firewalls between the different workbenches um for their own compute, their own catalog, uh and their own storage for persisting uh query results, create their own assets, and it gives them the ability to create their own products like dashboards um and whatnot.
[12:39] So, that's kind of the the crux of our our self-service strategy. Um and I uh real quick um just to give you a little uh idea of like a standard flow for AMI data um bringing in 5-15 minute interval
[12:55] um and event data from our meters flows into our head-end system um which then flows into our meter data management system. Uh we're actually pulling that data into the operational zone from our meter data management system um which is where deduplicating becomes important
[13:11] because if anybody's familiar with a uh traditional MDMS, um you'll have potentially multiple versions of the same meter read, estimated reads, actual reads. Um so, all that sitting in the operational zone, but then we um will bring that
[13:27] into our standardized zone and dedupe if you will so that you only have the the meter usage and it's an it's an important concept that meter reads and meter usage even though they might both be the same intervals are different things because meter usage is what the actual final actual
[13:44] usage of the of the of the meter was as opposed to all these different versions of the meter read. And then that flows into the actual product zone which then powers things like operational analytics like
[14:00] transformer load management or the CVR VVO is another big use case for us. So yeah, unlocks all kinds of business value through prioritizing investigations with uh uh
[14:17] let's just say non-technical line loss which may be theft or maybe misconfigurations. Um Our energy efficiency team loves being able to query across multiple different source systems to do their marketing and measuring the
[14:33] the effectiveness of their marketing. So unlocks all kinds of use cases and of course our regulator needs to see that we're being effective with our spend. Real quick, I'll go into one example. Um when we rolled out our AMI
[14:52] our technical go live which is middle of last year um we had an issue. There's a we're using SAP and there's a business process where it looks like it it looks at an hour a year of history for a given customer and
[15:08] estimates like what their next bill should be and if it's way out of whack creates a business process exception a BPM. And a CSR billing rep has to go and like investigate what the source of this exception is. Well, when we went live
[15:24] with the AMI meters, we had we went from like two register reads to four, and that actually threw off this logic and we're getting thousands of false positives, um which just like overwhelmed our our billing team. So, the AMI ops team was
[15:40] able to quickly go in and run some ad hoc queries and cook up a tool, a dashboard, by joining across device data, billing data, usage data, the interval data I was talking about, and of course BP data, and essentially
[15:58] filter out those false positives, which uh essentially saved the day for our billing team. Um And, you know, this is just a quick screenshot of what the analytics workbench looks like, and it's basically Databricks, that's nothing special, um
[16:15] but this is the an example of them being able to quickly go in self-service, um create some data assets that were gave them the ability to create a dashboard, which they were able to hand to our billing reps, and they could go
[16:31] in and just focus on the the the BP that were critical to investigate, and it ended up saving hundreds of hours of time. Um So, that's one example of the value of self-service strategy like
[16:46] this. And and then there's a whole list of different business outcomes that we're getting. I mentioned many of these already, the ability to in operational efficiency, which I just mentioned, grid reliability with our transformer
[17:03] load management use case. Of course, our customers love seeing interval usage, hourly interval usage instead of just one one number each month. And of course, the regulators, being able to give them the confidence that we're
[17:20] spending the taxpayers dollars prudently. So, I think the main takeaway that I want you guys to have is treat any uh uh any data
[17:35] opportunity like MI treat it as a product. Um Think in terms of products. Think in terms of reuse and governance. Um definitely give my opinion it's been incredibly successful to give our stakeholders and
[17:52] our business users, analysts, and scientists access to the data so that they can self-serve and and then measure it. Measure the results so that you can see what's working and what's not working. And
[18:07] this actually, in my opinion, sets us up for the what everybody's talking about here with AI. Um you know, traditionally garbage in, garbage out. So, this creates a trusted set of data in a
[18:25] with a context. Your ontology and your your data models that map to your business and that sets you up for being able to take it to the next level, which is essentially any of your
[18:41] new gen AI or agentic models to make your operations even more efficient. So, thank you very much.
[19:01] So, okay, it's working. Hi everyone, I'm Pedro, global head of data in EDP, data and AI director. And I'm bringing you here our journey, our strategy to turn data into autonomous operations. It's a slightly different approach and in fact it's
[19:18] another EDP, okay? It's not enterprise data model, it's actually Energias de Portugal and probably many of you don't know what EDP is, but let me give you an idea from Portugal, which is not a province from Spain, okay? It's a
[19:34] a sovereign country. We are actually the biggest company in Portugal. Uh and we are one of the biggest renewable energy companies in the world. We operate from there, but we have four regional hubs in four different continents, South America, North America, APAC and from Europe. And we
[19:52] have for a long time a deep commitment with energy transition. We choose heard. The motto is good. I think that someone interesting just repeated our motto a couple of months ago. We've been saying this for 2 years now, okay?
[20:09] Um we choose heard by changing tomorrow now and our commitment is actually to lead the energy transition. But with this kind of commitment, as you can see it brings a lot of challenges, okay? It brings a lot of internal dynamics
[20:24] changing, a lot of transformation happening. And it is actually happening in a moment of breaking moment actually for the energy industry. We need to adapt and we need to adapt fast. We need to rethink how
[20:40] the energy industry is working. And we can see that data and the AI technology it's actually mandatory right now. We are seeing it happening in several industries by doubling down the investment according to the EBITDA from 25 to 26, but it's actually even
[20:56] stronger in the energy and utility space. For that, okay? We decided that we needed to turn data into power and intelligence into action. It's actually written in our mission. It's written in our core, okay, of the
[21:11] company. And we established the vision to be data and AI leaders in the energy industry uh in the next few years. And we did that not only establishing a vision, but actually designing a strategy to achieve it. We designed the road map to ensure that
[21:27] we actually invested where we could create the most value so that we were not just spreading around, going into every single AI potential use case in the company. We did it by creating a real momentum of change in the company, and I'll explain
[21:42] to you what that means. And to build the foundations to be able to actually scale this up throughout the company in a consistent, governed, and repeatable way. Because we could do all of this, but without the proper foundations, the technology foundations in place, we could not do it anyways.
[22:01] Starting with the road map, it's a completely business-centered road map centered in seven big bets, seven value pockets that we could explore to ensure that we capture the most business value that we can from the technology. I didn't explain, but our
[22:18] our company operates both in the renewable energy asset space, energy management space, networks, clients. So, we have the full value chain of the energy of the energy industry. And we designed the seven big bets to ensure that we could capture the most
[22:34] value in each one of those segments, okay? And later on, I will bring you one of these examples of how we are turning data into actual autonomous operations. It's a concrete example. I think it'll be cool. Try not to fall asleep during the the rest of the presentation.
[22:50] But I think that the video will wake you up. So, I talked about the vision. I talked about the road map, but we also needed an operating system that would allow us to be consistent during this change. And what we did was to create an operating rhythm, okay?
[23:06] To align, to have strategically alignment, strategic governance to ensure the proper execution of the program, and actually the proper support, the sponsorship on every single line of the company to create this change. And we created and worked on the key
[23:23] enablers. And I'm talking about business alignment to have business close every time along the way, okay? We have a concept of one team where we put together business team, uh technical teams, delivery teams, and
[23:39] we ensure that everyone is aligned during the the execution of the road map. We decided on going on a federated approach for delivery to ensure the proper speed and the embedding of the AI change throughout the company. So, no central uh execution, no central
[23:56] definition, okay? Some central guidelines, of course, the way of working, but then delivery, it's close to the business so that we can ensure impact. And the adoption. And this one has been one of the key drivers of change on our company.
[24:11] To ensure the adoption, uh to ensure actual change in the company, we understood that we we needed to build the right skills and to come up with a proper training plan. And that means that we have we focus our efforts on three different roles, okay?
[24:27] The technology guys, the ones that are the ex- perts, the ones that build the change. We have the leaders. These guys need to sponsor, need to see the vision, need to actually help us change what is being done on an operational level. And they also are exposed every day to
[24:45] the AI transformation, and they are the biggest sponsors of this transformation. And every single employee. We needed to kind of break this vision that data and AI is some kind of evil that is coming to take all of our jobs and to help the people actually
[25:01] focus on creating more value with the technology, to focus the effort on added value activities. So, we created a program from e-learnings, self-paced e-learnings to workshops, a full week immersive with
[25:16] the teams, bringing challenges from the business and explaining how the technology works. And we provided them access to GenAI to every single employee on the of the company and then a premium layer for the actual guiders or the leaders of this change.
[25:34] And you can see with the numbers that uh in the previous slide I showed that we have like uh 11K employees and we have 22K enrollments in the in GenAI trainings, which means that we have on average two
[25:49] trainings by person on the company. And we are not even talking about what we call the priority group, which is the white-collar jobs, let's put it that the non-operational work. Cuz if we go there, we are talking about in terms of adoption between 80 to 90%
[26:05] of adoption every single day of the usage of the technology, okay? And this is only possible bringing this change, bringing this vision, bringing this kind of familiarity with the with the technology.
[26:27] So, vision, roadmap, operating change and operating rhythm, and now the foundations. We needed to have the proper foundations in place. Without the foundations, without one single platform, one single governance strategy to ensure data is trusted, to ensure data has quality,
[26:42] to ensure the scalability in terms of technology inside the company, we would not be able to do it. Operational analytics, we also looked into how can we improve our operations with the analytics
[26:57] and we built our own IoT platform to ensure that we could bring the data to the edge, okay? From the cloud, from the decision-making, from the uh let's call it a regular BI and machine learning um activities straight into the edge, okay?
[27:14] And the the example that I will show you then, it will show that. And also to have a trusted uh self-service platform to ensure the democratization of the technologies so that all the adoption that we have worked, all the training that we have done, that the people have actually the
[27:30] data at the point of their hands and that they can create value using this technology on a day-to-day on their work, okay? And of course here uh I mentioned to our strategic partner Databricks that helped us build this platform that we
[27:47] would allow us to actually focus on creating this value uh for the business with this technology, okay? On an open format, multi-cloud perspective, ensuring all the best practices are in place and that we are going with the best technology to do it.
[28:08] And uh I talked to you about the seven big bets. Uh now I'm going to focus on one specific example on asset performance in operations and I will show you it's not a pilot, it's actually something that is scaling throughout the company. Autonomous solar O&M.
[28:24] And this is actually one of our most interesting use cases, that's why I brought it here that is creating value to the company today, okay? It's not a pilot, it's not something that is going to ex- scale, it's not something that is in the lab, really fancy that we bring you here. No, it's actually something in
[28:40] production and that is scaling to at least 10 solar parks um Um right now, but I think that we'll go up uh below that during this year. Below that, no, above that during this year. Uh and what was our challenge? The challenge, I don't know if you have
[28:57] any idea, but the solar parks, the energy parks are actually usually remote, which means that the operation maintenance it's actually expensive. It means that we need to go there, we need to fix it, we need to see what's happening, we need to clean a panel.
[29:14] And that it's not safe cuz people need to drive there, need to reach there, and usually it's very remote. So, it has its constraints. It's expensive cuz you need to pay for that. And it's actually not very uh quick to do, okay? So, we thought,
[29:30] well, we have this problem, we have all these assets scattered around the globe, okay? And we need to fix this and to solve this, and we can actually do it. We just need to think it on a different way. And what we come up with was a platform to
[29:45] automate the decision process and to actually execute this uh operations of maintenance in the fields. We crossed um telemetry data, so real-time solar park data to understand what was
[30:00] happening on the on the parks. We crossed weather forecasts and real-time uh weather data to see what was actually happening in the park. We crossed with some metadata that we had, which has the characteristics where are the solar power panels, where is the
[30:17] park, what needs to be done. And then we deploy the drone to actually go and make the inspection and to understand what was happening, what needs to be cleaned, what needs to be sued, okay? So, that after that, the drone comes to
[30:32] the to the platform and gives the instructions. It's all supervised, for now at least, gives the instructions to the the autonomous vehicles that actually go and sue and clean and do all of these autonomous. Okay?
[30:48] It's actually the first one was a solar park in Spain and just in a pilot phase we saw 500 issues detected, more than 100 hours of of activity from the autonomous vehicles in the first few months. We are right now having uh this
[31:04] technology installed in parts in Spain, Romania, Italy I believe, and we are thinking on how to do it also in the US. This And when we looked into the business case, it actually exists, okay? It delivered value. We are talking about
[31:20] uh we when we go full scale to the initial scope defined for the project, which were around 20 parks to 30 parks, we're talking about more than 1 million of impact in a beta on a 5-year time frame.
[31:38] So, this was the solution that we are excalating uh everywhere. Um and it is actually one of the examples, one of the examples, okay? Of how we are every day rethinking how we do it, rethinking how we deliver more value to our clients,
[31:54] and rethinking how to use and embed this technology on our day-to-day operations. Let's see if it starts, and I'll just show you uh a cool video. It was not done for me by
[32:09] me, of course. Um on uh of the technology working so that you have an idea and that you don't think that I'm just bringing something very interesting but it's not working. Uh so that you can see it actually functioning. The energy transition is one of the
[32:26] greatest challenges of our time. At EDP, investing in automation and robotics is accelerating this shift. With these technologies already transforming how we build and maintain energy infrastructure. By using
[32:42] cutting edge robotics and drones, we can construct, inspect, and optimize energy production faster, safer, and more efficiently. From automating large-scale installations to real-time robot inspections, these innovations reduce
[32:58] construction time, cut emissions, and improve operational efficiency, all while ensuring a safer working environment for our teams. It's not just about technology. It's about scaling automation to help the photovoltaic industry grow. When innovation meets teamwork,
[33:15] we unlock new possibilities and move the energy transition forward together. Right choices can take innovation to new heights. EDP
[33:30] So, actually shows that data and AI is no longer a support capability for the energy transition. It is actually a core operating capability that we need to leverage to ensure that we can deliver our our mission. Thank you very much.

Learn more about the Databricks Data and AI platform.