From Azure Synapse to Databricks: Alinta Energy's Enterprise AI Platform Modernization

Summary

Alinta Energy, an Australian electricity gentailer serving over one million customers, migrated from Azure Synapse to Databricks over a seven-year journey, consolidating more than 90 source systems and 15,000 data products while achieving 139% consumption growth year-over-year with flat platform costs.
Five core practices define Alinta's high-performing data team: team design before technology, embedding data investments in business projects, mastering DevOps and platform engineering, implementing governance for confidence rather than compliance, and making deliberate trade-offs about what matters.
Alinta built ACE (the Alinta Intelligent Strategic Engine), a multi-agent system combining 500 structured metrics and 9,000 enterprise documents for executive decision-making, with confidence scores and direct source attribution on every answer.

From Azure Synapse to Databricks: Alinta Energy's Enterprise AI Platform Modernization

Alinta Energy, Australia's leading electricity gentailer, has transformed its enterprise data estate on Databricks, consolidating over 90 source systems and 15,000 data products while supporting 36,000+ daily queries. The platform generates cost-neutral growth: consumption increased 139% year-over-year while platform costs remained flat, demonstrating how disciplined platform engineering drives sustainable value.
General Manager Brad Walker shares five core practices that define high-performing data teams: prioritizing team design before technology, embedding data investments in business projects, mastering DevOps and platform engineering, implementing governance for confidence rather than compliance, and making deliberate trade-offs on what matters. Learn how Alinta built ACE (the Alinta Intelligent Strategic Engine), a multi-agent system that brings together 500 structured metrics and 9,000 enterprise documents for executive decision-making, complete with confidence scores and direct sourcing.
🤝

Chapters

FAQs

How did Alinta Energy achieve cost-neutral growth on its Databricks platform?

Alinta grew data consumption by 139% year-over-year while keeping platform costs flat, which this video attributes to disciplined platform engineering and FinOps practices embedded in the team's culture. The General Manager of Data and AI describes this as evidence that operator discipline—not the technology alone—differentiates high-performing data teams.

What is ACE and how does it support executive decision-making at Alinta Energy?

ACE, the Alinta Intelligent Strategic Engine, is a multi-agent system built on Databricks that brings together 500 structured metrics and 9,000 enterprise documents, giving executives a single interface for querying operational and strategic information. This video explains that ACE provides confidence scores alongside each answer and links responses directly to source documents so executives can verify the basis for any recommendation.

What are the five core practices that define Alinta Energy's data team?

Alinta's five practices are: prioritizing team design before selecting technology, embedding data investments directly in business projects rather than IT budgets, mastering DevOps and platform engineering, implementing governance as a source of confidence rather than compliance overhead, and making deliberate trade-offs about what the platform needs to do well. This video presents these as the principles that sustained a seven-year Databricks journey.

Why did Alinta Energy move from Azure Synapse to Databricks?

Alinta began with a small Azure Synapse environment in 2018 that was struggling to scale with the company's growing data needs. After commissioning an IT strategy that identified cloud modernization as a priority, the company migrated to Databricks and has since consolidated more than 90 source systems onto the platform. This video describes the move as part of a broader shift from a traditional on-premises SQL Server data warehouse to a modern, scalable data and AI platform.

Full transcript

[00:08] Well, good morning. Uh I'd like to thank Gavin for what he just said, but I couldn't hear it. So, uh I hope it was good. Um So, today, uh very impressed that you've all managed to turn up after last night. I hope you enjoyed yourselves as much as we all did.
[00:24] Uh and we're just going to have a little bit of a talk about how we actually uh make Databricks work for us. Um so, my name's Brad Walker. I'm the general manager of data and AI. Um I'm from a
[00:39] small company in Australia. We run uh we're we call ourselves a gentailer. So, we generate, we trade, uh we retail to over a million customers. And uh we've been we've been on this journey with Databricks for uh 7 years. So, uh let's get stuck in.
[00:57] So, everything's about AI these days. And the interesting part is the hardest part about making AI real is not the technology. It's the discipline that you need to put into bringing it to life. Uh and when I talk about discipline and operations, that's very much coming from
[01:14] the world of uh heavy industry and power generation. So, uh I'll just set the scene for you. It's 3:00 in the morning. Uh a critical data feed from a source system has just failed. And my on-call support team are happily asleep in their beds. Why is that?
[01:32] Because we've built every pipeline with a very high degree of resilience, because that's what is expected of you when you work in the energy industry. Uh the day we started providing our data into our enterprise platforms
[01:47] supporting our power stations, running our trading, was the day we started getting measured the same way as the operators do in the control room and on the trading desk. So, when you work really hard to get your data platforms to be a key component of a business and
[02:03] right across the spectrum, uh the operator's discipline is what I believe differentiates you from everyone else. So, back in 2018, uh Alinta Energy commissioned an IT strategy. And that
[02:20] was uh that was fantastic because it set the foundation for a complete cloud modernization and the understanding that we needed to do something different with our data. You know, the traditional on-premise SQL Server data warehouse uh was not the way of the future and we
[02:37] had at that time we had a small Azure Synapse environment, which was struggling to scale as well. So, in 2019, I started with Alinta and we wrote a data and analytics strategy as I'm sure most of you have. Um and one of the
[02:53] key things for our data and analytics strategy was how do we how do we actually build this? And we started with the team. So, today I'm going to take you through the five practices that I think differentiates a modern team running one
[03:09] of the world's best platforms. So, as you would not be surprised when you build a strategy, all everybody wants to know is what platform are we buying and which SI is going to do the
[03:26] work. But we didn't start there. We started designing and building the team and inventing the operating model. And the most important part about the operating model was how we going to collaborate to create value from data. So, before we did any of that, we
[03:42] designed the team and it's a fully integrated team. It does strategy, architecture, delivery, of course data engineering, DevOps engineering, platform management. So, all in one team. Oh, I forgot data
[03:57] science. How we How could we forget the data scientist and the machine learning engineers? So, team before technology, fully integrated. Um and just to give you an understanding, uh the platform that I manage has grown
[04:12] to uh we've got over 90 source systems, 2 and 1/2 thousand source files, source objects coming in, uh upwards of 15,000 data products across the across the estate. Um and I've done that with a team that still
[04:29] fits around a large boardroom table. We're a team of 20. So, uh I think I think the most important part is to ensure that you leverage the platform, not head count to scale. And one of the things that's really interesting is what we've seen here this
[04:44] week is the continuing trend of augmentation with those fantastic Genie products, uh especially Genie code. And uh the Genie code ML is looking very good. One of my
[05:00] lead data engineer said to me when I was coming here, I said, "Josh, what what makes the team different? What Why are you here?" And he said, "Uh the team is a group of free thinkers, tinkerers, and the endlessly tech curious." So, one of the things that's really established a
[05:15] culture of mateship and collaboration is that they all get to roll up their sleeves and do some of the most amazing and modern data engineering that uh the world is doing. So, uh it's something that is very enjoyable and it we've found that the team are very very happy
[05:32] in their roles. We haven't We We have very low churn, and we have uh a pretty long list of people who are interested in starting with us. So, if there's anything to take away from today's talk, it's the team that matters.
[05:52] For of you who are having fun during 2020 when COVID struck, we were adjust into our first phase of our transformation journey. And because of the nature of COVID, the Australian energy sector was responsible for taking up all of the risk of people not being able
[06:08] to pay pay any bills. We we had to ensure that we had enough capital on hand to to keep the retail customer base happy cuz none of us knew what would happen. Uh inevitably, that meant we lost all our transformation funding. So, we could
[06:24] have stopped then, but we were very lucky. Our CIO and CFO worked together with the rest of the exact team, and they actually decided that every business project would carry with it investment in our data and AI platform. So, we went
[06:42] from a standalone group trying to get the attention of our execs and our business teams to a group that was delivering with them on every one of their projects. And just an example of some of those, we built our own billing system for over a million customers. And
[06:57] part of that was an integrated suite of regulatory report and operational reports and deep analytics for our retail business that we built in tandem with the billing system team on Databricks. Uh we worked very hard on many regulatory
[07:13] reforms, many changes. Our our industry in Australia is one of the heaviest regulated electricity markets in the world, and we regularly have to update our capabilities about regs. So, reg reports did a huge job for us bringing in a lot of the systems that we
[07:30] needed, and that was a great justification. And then customer initiatives, so the chance to actually build out the data science environments, the machine learning capabilities, and to build out the data products that would go on to run our retail business. So,
[07:47] probably the takeaway on this one is that what could have been catastrophic for us actually turned out to be a fantastic operating model to carry us forward. And to this day, we don't have stand-alone funding. We work with the business, and they justify the
[08:03] funding uh when we need it. So, it uh turns turns your customers into your advocates, which has been fantastic.
[08:20] Now, a topic that I'm rather excited about. We started a journey about 3 and 1/2 years ago to really transform our DevOps and platform management. Uh we were migrating off Synapse. We had a Synapse front end for our end users, and we'd
[08:36] invested heavily in Databricks and the lake. Um and this was when the lakehouse started to be a real thing. So, uh we had the evolution of SQL warehouse and the opportunity to get rid of our Microsoft Azure, uh which I must say was a very rewarding
[08:54] thing to get done. Um we had an outage uh in our data centers in Australia a few years ago, and my Synapse environment at the time uh took 4 days to recover. My Databricks environment sitting in the same tenancy, in the same
[09:11] data center, took 3 hours to come back. So, for us, Microsoft was not a trusty trustworthy platform, and so getting to a full stack on Databricks was very important. But, one of the things that I'm sure those of you who have got mature Databricks environments knows is
[09:27] that Databricks is not the platform you buy if you want to do your architecture with crayons. If you choose Databricks, you need to be competent at managing the stack and getting the most out of it. They're doing some phenomenal jobs with serverless, but at the end of the day,
[09:42] my DevOps team can stand up a new environment purely from infrastructure as code. Uh so, recovery, restoration, innovation, new environments is critical. So, we've spent 3 and 1/2 years investing heavily in our DevOps practices.
[09:59] Um and year-on-year, the investment has paid for itself as well as adding features for our engineers. So, the probably the best thing for me is that our CFO loves us. We've been growing the platform year-on-year. Uh in the last year, we increased the
[10:16] end-user consumption by 139%, but for 3 years straight, we have been flat-to-down on platform costs. I haven't met anyone else here who's had that joy.
[10:32] Um and the part for me that is really quite exciting is conversations with the CFO are no longer the hardest ones you have in in the day. They They genuinely uh appreciate what we do. So, the key takeaway on this one is invest
[10:49] in your DevOps, invest in your FinOps, every day run the gamut, measure what you what your users are doing, and then spend time with them tuning, optimizing, understanding why they've been putting these crazy workloads on new clusters, and that's
[11:06] been a joy for us. So, DevOps has been a extraordinary success story for us. A topic that has actually got interesting with the investments
[11:21] Databricks have made in Unity Catalog used to be a very dry topic that very few people wanted to talk about. Um I've had the privilege of working on a number of very large uh uh master data management projects and other cataloging and governance
[11:38] initiatives. And one of the things that uh we learned is that as the rush to AI comes on, the challenge is never the technology. It's trust in what we're doing. Um Unity Catalog for us is fantastic. It has
[11:55] enabled uh the scrutiny of the data, the lineage, the controls. Uh but Unity Catalog alone will not get you a pass at your audit committees or your risk councils. They need to trust the process and trust in what you're doing. So, we introduced
[12:11] a number of uh very clear uh AI policies and governance practices and wrote all of those in very plain English so that the people outside of the data community could actually understand what we're doing. So,
[12:27] the fantastic part was the platform enabled us to have certainty about what we're doing, but the engagement with the communities is what gave us the privilege to continue. Now, I don't think anyone's been allowed to
[12:42] come up and speak today if you didn't have something about AI that you'd just done recently that you were kind of proud of. And I must say, coming from Australia, I've never been surrounded by so many people who have done fantastic things with AI. So, it's a very very rich
[12:58] uh amount of activity that's going on around us today. And something that we recently released that we're proud of uh is the is a platform called ACE, or the Alinta Intelligent Strategic Engine. You've always got to have a good name.
[13:13] Um I'm not sure if that's one, but we like it. Um and the interesting part about ACE is it came from a collaboration between our CEO and our CIO. And uh the conversation came to us that uh they thought we'd invested heavily in our
[13:29] platform over the last six and a half to seven years. So, they wanted to see what it could do for the executive team on the AI front. Uh so, we've collaborated with Databricks and over a period of four weeks
[13:45] with about 70 person days, we managed to build a multi-agent system that brings together 500 of our structured data metrics and over 9,000 documents that the board and the executive team rely on.
[14:01] Board papers, strategy decks, market research. And our system leverages a supervising agent that decides what is the nature of the question being asked by the executive and then returns the result.
[14:16] The thing that I think's a little different is it returns a result that has a confidence score and it tells them you can make a decision on this. It tells them, this is directional but good for deci- for your thinking. Or this is generally correct but don't bet
[14:33] the farm on it just yet. And so, it also links to the specific document or the specific metric and it can surface Genie, AI/BI, or it can actually take them to the document that the insight was from. So, it's no longer an idea that our CEO
[14:51] wanted. It is now an actual tool that our executive suite through our C-suite are using. And the best part is they're all nominating people from their teams to build out the agents that will represent each of their business units going forward.
[15:18] So, the operator's discipline in my experience, I've been running enterprise data warehouses and data and AI platform for the 26 years. And this is why I said at the start, the team is what matters and how you approach and collaborate with the business. So, first, focus. Build the team before
[15:34] the technology. Compound. Every opportunity you get to invest, build out your estate. Build out the quality of your sources. I currently have over 90 source systems for a company running generation trading and
[15:50] retail. That's a lot of systems. Platform engineering is absolutely a differentiator. It is the only way you'll continue to get the funding you need to grow the business, and we all know the pain of tokens. So, managing the cost of compute and storage is the
[16:06] last frontier that you might save any money. Governance. All of our governance is about giving people confidence that they can use the data in our platform, that they can take action, that it represents the facts of their part of the organization. So, build governance for
[16:22] confidence in your customers, not to create a stack of paper that looks like the auditors have just been through. And the last one that I think is really important is being fierce about what matters and honest about what doesn't.
[16:38] When you're trying to build an enterprise-wide data and AI platform, there are some decisions that just have to go the right way. There are many things that don't have to be perfect or that don't have to go the way the IT team work. A lot of the things we do, we do because it meets what the business
[16:55] needs at the time they need it. But sometimes you just have to make sure you get the right outcome, or you'll be stuck with tomorrow's legacy. So, five practices.
[17:18] I think that by the fact you're here, you've either picked Databricks or you're thinking about it. I picked it 7 years ago and I couldn't be happier. It enabled an architecture that I'd literally been chasing since the year 2000. We finally have the technology that does everything a data platform could ever
[17:34] want to do. The evolutions that we're seeing here today are just creating a monstrous amount of work for us when we get back home, but there's so many cool things to put in. But you know, pick the best platform and then bring in the discipline and earn what it
[17:51] can do for you. In summary, we didn't start with an operator's discipline. We were IT people. We We barely understood the real world of operations. So, I think
[18:06] we started from a strategy and that first decision to invest in the team. But everything grew from there. One project, one engineer, one governance call at a time. The exciting part we've uh we've spent
[18:21] quite a bit of time removing legacy platforms and this year we're turning off the last of five legacy platforms that we've uh been able to get out of our environment. So, we're now at the point where we've got Databricks wall-to-wall from power station to boardroom. Over 90 source systems.
[18:39] About 40% of my business use my platform directly. So, we've got deep penetration and it's all come from the fact that we decided that if we wanted to earn the respect of our colleagues who ran and kept the lights on, we needed to do a
[18:54] decent job. So, the platform's got all the tech you need. The discipline's what powers it. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.