Automating Engineering with AI - LLMs in Metadata Driven Frameworks

Summary

• AI-Driven Automation: AI is revolutionizing data engineering by moving from code generation to config generation, enabling teams to automate repetitive tasks and reduce manual work.
• Metadata-Driven Frameworks: Metadata-driven approaches with AI enable data teams to focus on strategic work rather than optimization, transforming how data pipelines are built and maintained.
• Industry Disruption: Data engineering as currently practiced will change fundamentally; automation will shift engineer focus from system optimization to delivering better business outcomes.

Automating Engineering with AI - LLMs in Metadata Driven Frameworks

Talk by Simon Whiteley, CTO, Advancing Analytics
The demand for data engineering keeps growing, but data teams are bored by repetitive tasks, stumped by growing complexity and endlessly harassed by an unrelenting need for speed. What if AI could take the heavy lifting off your hands? What if we make the move away from code-generation and into config-generation — how much more could we achieve? In this video, we’ll explore how AI is revolutionizing data engineering, turning pain points into innovation. Whether you’re grappling with manual schema generation or struggling to ensure data quality, this video offers practical solutions to help you work smarter, not harder. You’ll walk away with a good idea of where AI is going to disrupt the data engineering workload, some good tips around how to accelerate your own workflows and an impending sense of doom around the future of the industry!
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How is AI changing data engineering?

AI is shifting data engineering from manual code optimization to intelligent automation of configuration generation. This enables teams to work smarter by automating repetitive tasks and letting AI handle complexity while engineers focus on strategic value.

What is the difference between code generation and config generation?

Code generation asks AI to write code, while config generation uses AI to generate configurations that systems understand. Config generation is more efficient and maintainable for data platforms like Databricks.

What pain points does AI address for data engineers?

AI helps engineers overcome repetitive manual work—schema generation, data quality checks, and workflow acceleration. It transforms pain points like growing complexity and time pressure into opportunities for innovation.

Will automation eliminate data engineering jobs?

The fundamentals of data engineering—helping people make better decisions with data—will remain. However, the nature of data engineering work will change as routine optimization tasks are automated, requiring engineers to adapt and focus on higher-value problems.

Full transcript

So uh I kind of changed this talk around a little bit from when I first wrote it. The first thing I did was looking at a very specific thing going here's how you can use AI to do this one thing in the world of data engineering and then kind of iterated with all the announcements this morning at the keynote and everything going on. I've kind of switched it to a bit more of a not necessarily a existential terror about the future of the industry, but more about a how you should be thinking and positioning yourselves to get ahead of all the stuff that's happening right now.
It's kind of where we're going. Normal stuff. Data bricks refused to be sued for anything I say. Do the surveys. It's always useful. And yeah, I thought I'd start with a non-controversial statement just to chill everyone out for a moment. Uh, and this is this is an interesting one. I've had this conversation time and time again with data engineers. And it's not even AI coming in and stealing the stealing the work, right?
It's things like serverless, things like data doing predictive optimization. It's the maturity of what we actually do and the platforms that we use are slowly taking away the things that we do as engineers, right? They're taking our jobs. But actually, what do we do as a data engineer? What is our fundamental purpose? And I keep having conversations with our engineers and they're like, I make Spark go fast. It's like, that's not your job.
No, no one no one needs you to do that. Yes, we currently have had to do it, but your fundamental job is to get people to make better decisions based on their data. It is not to optimize a Spark pipeline. It's not to make your delta tables as efficient as possible. We have to do that because the technology needs it to, but that's not why we do it. We do it for the end user. We do it for the end result of things. And that is still going to be a thing that we need to do.
But the amount of work that we currently do to make stuff actually work, to get data from point A to point B to put it in someone's hands so they can make a better decision, that amount of work is going to shrink. We're going to be able to do more. But we need to be able to be on top of it because if we don't, and you know, if in three, four years time you're saying what I do is I make Spark go fast, companies are going to go, well, we don't need you to do that.
So it's not data engineering isn't dead but data engineering as we know it is going to fundamentally change over the next three four years and what we need to be doing is just thinking how do I prep myself now to get ahead of it so that when it does change I'm not suddenly left there going oh no one no one wants to pay me anymore so kind of that's where we're going that's the things I'm trying to get ahead of essentially you do not want to be a ly everyone familiar with the industrial revolution you know the whole ideas of you don't want to be the guy that was actually handmaking clothing a card hand you know kind of what I do is I sew a jacket someone comes along with a loom and says I can make 10,000 jackets in the time that you can do one that person doesn't have a job anymore and then fast forward you know even the digital age the digital revolution you had people saying I am fantastic at mental arithmetic and then someone else goes well I've got a calculator like why do I need you like I was forced in school to do you know I had my calculator exam, my non-cal exam.
I had to go and do one where I had to work it all out. They don't do that anymore because you don't need to. Everyone always has a calculator on their phone. And that is where we're going. We are now at this point of the AI revolution where if you are the data engineer who isn't using any AI, you are going to find yourself working so much slower than literally everyone else. And that's uncomfortable. And every conversation I have with a data engineer going, "Are you using AI?"
They go, "No, I will not be replaced by AI because AI can't do my entire job." And people see it as a very binary onoff. They say, "Can it do everything I can do?" No. Then I'm safe. My job's fine. But that's not what it's about. Can it do 20% of what I can do? Great. Well, then I can do loads more work. Can it do 50% of what I can do? Yeah, I can do more work. It's about finding the slices, the tasks that we do as data engineers and say, "How can I make myself more efficient by I don't need to do that anymore.
I don't need to optimize tables anymore. I don't need to design cleaning rules anymore. I don't need to write data pipelines anymore. More and more this is going to increase and increase and increase and we're going to take away this work of what we do. But if we're not doing it, someone else will. And in four, five years, you'll find yourself going, "Oh, I'm turns out I'm now out of date. Turns out I don't have the skills anymore.
Turns out I'm not doing it. But it's a dangerous trap of going unless it can do everything then I'm not going to use it. And that's kind of where data engineering is currently. People are scoffing going I don't need AI. It's not going to be right. I'm going to if I got AI to design all my cleaning rules it's going to be like maybe 70 80% accurate. And I get all the engineers going well if it's not 100% accurate I'm not going to use it.
It's like why not? If it can save me 80% of my time and then all I need to do is the adjusting and get it right. Suddenly I'm competitive. If I can do things faster, I can do more. But we're on that we're on that turning. We're on that cusp of when people are starting to challenge what we're doing. So, oh god, I hate it. But we're in the era of vibe engineering. We're in the era of people who actually don't haven't been data engineers, don't understand the Python, don't know what they're doing, but actually can jump on chat GPT, can use data bricks assistant and go write me a pipeline that does this.
And it'll generate a code and they'll use the code. And then if the code breaks, they'll get the AI to fix it and they will actually through brute force get the data from A to B. That is happening now. Now it's a little bit early. It's in like early adopter stage, right? But give it a year and we'll be using so much of just AI generated code for everything that we're doing. We kind of need to be getting ahead of it and going how can I make sure that doesn't go wrong.
Fully automated performance is just happening right now. It's going to get more and more serless is doing all that stuff. AI based lift and shift migrations. You saw some lake bridge things happening this morning. Like basically saying, I've got a thousand store procs over there. Make that into a thousand data bricks notebooks. That exists. People are using it now. And if you're there talking to, you know, your senior execs going, "Oh, migrating that system.
Oh, it's going to take me a year because I've got to do all this stuff." Someone's going to come along and say, "Well, I'm just going to automate it. I'm going to push a button and it's going to translate over." That stuff is actually getting to the point when it's pretty good. And that is new. And then fast forward a little bit, you know, ETL you saw this morning the yeah just load this data, clean it and put it into a model and an agent will go through and do it.
Now is it good? But we are brand spanking new there. Give it a few years to mature. You're going to see it used more and more and more. People are going to be doing this like refactoring and things as we go. It's going to happen. That's the I don't want to lift and shift. I don't want to take a thousand store procs and turn that into a thousand notebooks. I want to take a thousand store blocks and just migrate it into a metadata driven framework.
You can build that now but people aren't doing it yet. This is like the thing people are suddenly looking at and then yeah what four four plus years data is just going to be autonomous. That's like a pretty wild prediction, right? But the things you see this morning, this is where things are trying to head and it's going to be chaos. It's going to be utter chaos about how we do it. So, how should you guys be thinking of it?
If you're a data engineer now and you're looking at all this stuff happening, what can we do? Is it just inevitable? You just give up. Go open a bakery and give up on life. You can actually start trying to get ahead of stuff. Um, essentially what you'd have to be doing is thinking of AI, thinking of agents as another skill, another thing in your tool belt. So, who who here is currently a data engineer? Who here currently writes AI?
Oh, okay. better mix than I thought. Probably because I had AI in the title of the session. If I went to just a performance tuning thing and got all the data engineers, the majority of the ones I speak to don't do any AI. They're like, "No, that's an AI job. That's a data scientist job. It's not for me. I'm a data engineer." And like that vin diagram, there's very little overlap for what we see. And it has to improve the idea of AI literacy in everyone.
If you are in the data space, it's no longer data and AI as two separate fields that are kind of slightly joined. AI is just becoming absolutely everywhere as you've probably got from the conference. Literally every thing they announce is oh and this uses AI. It's just what's happening. I mean this is something I stole a little bit from Gartner. Uh Gartner had this idea when they're talking about like AI strategy when you're doing a what kind of use case do I have for AI?
They have this is it for an external facing is it customerf facing? Is it something that's going to make the company money? And then they have like the internal efficiency idea. And this idea, is it a little incremental improvement or is it a big groundbreaking fundamental shift in what we do and almost everyone you speak to when you're talking about AI, you talk to your data scientists, you talk to your AI guys, you talk to your co about the strategy, they focus on the top bits.
They're like, "Yeah, we're going to put something so our customers suddenly get a better experience. We're going to develop products faster. We're going to get big chunky things that I see an incremental uplift in revenue." What I barely hear from anybody is an engineer going, "Oh, I'm just going to use AI to automate what I do." And the platforms are going to start doing it. We as engineers need to start doing it now so we're actually ready by the time it happens.
So that's small incremental improvements, but also the major shifts is things that we can actually start doing now to get ahead of things. So there's loads of different ways we should be using AI as an engineer. Like there's the basic stuff, the absolute basic stuff, right? There's the I should be doing meeting summaries. I should be writing stories using it. I should be building acceptance criteras. Now, I'm not just blindly accepting it and going, "Yeah, yeah, that'll be right."
But actually, anything that I spend my time on, I should just be trying to make more efficient. Then, as we get down, we can do so much more. Now, AI augmented engineering is a whole idea that we just don't see enough people doing. So, you know, the basic stuff, right? Yeah. Work efficiency. I should rock up to a meeting. I should let AI take the notes for me. I should let AI actually work out the actions. It should do the summaries and follow-ups.
I don't need to do that anymore. That used to take me hours. I'd run a discovery session, run a big workshop with a load of clients, and then spend the rest of the day writing it up. I don't need to do that anymore. I've saved myself a ton of time. Putting comments in your code, doing the pull request reviews, all of that stuff. Yeah, it's been a bit rubbish for the past couple of years. It's now getting to the point where it's actually quite good. we should be embracing it and using it essentially where you spend your time the more you can carve out and go well that's not actually my job it's just a thing I have to do if we can carve that out great but that's not hard that's you know I can go in teams I can get teams premium I can do summaries that is all just commodity stuff that's just being infused into what we do if you're not using it you should be asking yourself why am I not using it yet that's easy coding assistance yeah everyone's using cursor vs code databicks assistant writing your code with a little AI sat next to you going where do you write this a bit better should be using it it's just making things faster who doesn't use some kind of AI coding assistant currently exactly see all fine but with the danger of going you know what it's doing you know why it's doing it you can look at the code that it generates and you can say is that right yeah okay I know what I'm doing that's generated some right code I g the super quick example of asking it to generate a little bit of streaming for me go and generate me some streaming code it writes some god awful full code with a little bit of a hint going that I don't want to just that no that makes no sense.
The config should always be going in as a quag that I then extrapolate from. That's fine. But that takes I need to know what I'm asking for, right? I had to adjust my question rather than going just write me some streaming. I had to go write me some streaming but do it this way. I still had to know what to ask for currently. Now they're going to get better. It's going to improve. But as an experienced engineer, I can kind of nudge it and guide it.
I mean, that's still not great code, but it's better. But all those things is just efficiency. But that is literally the minimum, the absolute bare minimum as a data engineer to be current and relevant today. You should be using those things. That is like the baseline entry. And that's shifted so much in the past couple of years. There's like a baseline of AI literacy that is just the you need to be this tall to ride the game, right?
But that's changing. So the other bits the more actual how do I use it as an engineer lift and shift we're seeing a huge amount of so that kind of migration approach of going translate from one language to another we're going to see with the idea of lakebased coming out yesterday a load of OLTP stuff being translated automatically into the postgress stuff to put it into data bricks we're going to see lots of people just migrating from systems to the point when doesn't really matter what systems it's in you can just convert between languages fairly easily that's becoming almost trivial But as I said, if I've got a thousand store procs, I don't want to convert that into a thousand notebooks somewhere.
That's terrible. And that's not even the worst way it's going to get. But we can get ahead of that stuff. So digging into it actually a bit further. So what do I do as an engineer? What is my endto-end ETL journey? Well, I'm looking at to how do I augment my data? How do I do some how do I look at a source system and extract data from it? How do we do data cleansing? How do I look at some data and go, "Oh, I should remove special characters from there.
I should normalize that date. I should transfer currencies." All of the stuff that we do as engineers is actually becoming something we can just automate. Never really do it properly before because it needed some decision- making. How do we design a schema? How can we actually say this is the data model it should use? And everyone goes, "Oh, no. I don't want it to design my schema for me." But then, I'm sorry if you're like in insurance and you're designing a claims model.
You're not the first people to do it. It's getting to be actually fairly predictable. Most data schemas have probably actually been designed before. Now, not to say it's going to be perfect, but you can get 80% of the way there and there fit it. Source target mapping. I've got these silver tables. I'm mapping it to this data model. Just generate the logic. That works really easily. And again, when we were first looking at it, we get that kind of the scoffing of the data going.
It's not going to be perfect though. and it's the immediate dismissal of unless it's perfect, we just don't use it. But it's gonna become commodity and we're going to have to start getting on board now so you're ready for it when it comes. So there's just lots and lots of stuff going on in that kind of classic ETL journey. And that's the mistake. The mistake is assuming that it's you push a button and it does it end to end and if it doesn't do it, it doesn't work.
Whereas the idea is pulling out going each step, each step of this journey. Can I make that faster? Can I make that faster? Can I that faster? and slowly getting that acceleration improvements. If you build something just as the first two, it's going to look at a SQL server, scrape out the metadata, build out the config that you need to generate it, and apply some cleaning rules. That saves you so much time. Imagine if you talk to someone in the business and they go, could I start doing some prototyping on that data?
You're like, no, we haven't on boarded it. We've not connected to it. And it takes you what, an hour or two to get that loaded and to get that cleaned. That is massive. Whereas currently that's what at least a sprint couple of sprints like massive incremental improvements about how quickly we can do stuff and that shouldn't be a oh my job's going away it's a I can do so much more the business can do so much more we as engineers can do so much more so it is it is growing massively now there's a a middle bit which is going to be awful I mentioned that vibe engineering stuff yesterday in the keynote though talking about using agents agents are going to be generating code and it's going to be massive and we're going to see this absolutely all over the place.
If I'm loading a thousand tables from SAP and I want to have that loaded and cleansed and I give it to an agent to go load and clean that data for me, the way people are currently thinking about this is the agent will generate the code and it will generate a separate pipeline for each of those tables. I'm going to have a thousand slightly different data loading pipelines. anyone else just full of fear at the prospect of that.
A large language model is a sarcastic model. There's a probabilistic difference in terms of the thing that it generates. The pipelines are going to be very slightly different. Now, currently that's not good enough because we can't quite trust it. It's very new. It's immature. Maybe in the future we don't even see that and we don't even care. Who cares if the pipeline was written slightly differently as long as the data arrives in the exact state I wanted to.
Does it matter? Currently it matters. It's not going to matter in the future. But for now, that's the idea that people are trying to use this stuff, but they're using it as a code generator. Like back in the day of Microsoft ETL, we had things like Biml and Wescape. We could like put some metadine, turn a crank, and it would generate SSIS packages or SQL for me. But I had this massive code footprint of a thousand generated procs I then have to look after in production.
That's not the way we should be working. So the way to think about how do you use it now before this stuff becomes commodity in this next one to two years if I've got some metadata driven framework if I've got a little bit of JSON config YAML config wherever I happen to choose to do it saying load this bit of data here's some transformations here's some cleansing however the hell I've decided to actually build that thing then I've got a bit of dynamic pispark saying run this same set bit of pispark a thousand times in parallel for this one use that bit of config for this one use that bit of config that's how we're building metadata driven frameworks today, right?
We're not generating the latter code. We're generating the config. So if you're building things now to go, well, just scrape all that information. This is the syntax that the config should be in. Go and generate all my cleaning rules. Your actual code isn't changed. You can unit test your code. You can work with the code. You might use a coding assistant to write that framework really quickly. But actually, we can put a line in the ground.
We can put a peg in it and go, we know our code is fixed. We know our code is good. generate all the config because that's the bit that takes time. But we're not seeing people do that yet. That is where we need to be thinking now before it gets to the everything's just an agent and no one cares anymore. It's this whole idea of what we're doing. And I did a the quickest little demo. Like it's something I built last year of just jumping into chatgpt.
You wouldn't do this in chatbt. You wouldn't send your data into a random third party thing. But this is something I took five minutes to do. Imagine if you take this and you productionize it. You make a local LM. You build an agent into it. Just saying you're a data engineer. Create some rules. Here's some existing Python functions that are in my framework. It wouldn't know what's in my framework. I'm having to give that as part of the prompt context.
Here is an example of what the empty contact looks like. Here's an example of what the transform looks like for a different thing. Right? I'm giving it enough context on the fly. I'm having to do it manually in my one big prompt because I'm just cheating and doing a quick PC. But you would do that as a straight lookup from your source code. You could do it as something you do you inject into your prompt as part of your agent.
Give it some information. In this case, I've just said, look, here's the schema. I could pull that from Unity catalog. I could pull that by doing a quick data frame inference. I could give it a sample of my data if I wanted the large language model to have some of my data. In this case, it's not going to be as accurate because I'm just giving it the schema. But what you can do with LLMs these days, you can say, here is the fixed output I want you to do.
I want you to always give it me in this JSON schema. I can make sure that's predictable. It's not going to hallucinate. It's not going to have variance. It's always going to do a fixed schema. That is just a thing you can turn on. Now, so in this case, it looked at it. It generated some cleaning rules with some justification. It's generated my contract for me. Like saying, hey, I'm loading a thousand tables. Just decide what cleaning you should do.
I still need to review it, but that saves me a massive amount of time, right? That is a gamecher for how quickly we as engineers can work. Oh, I'm having a light flash at me. I've got maybe two minutes left. We'll finish things off quickly. But that is how we should be thinking. We should be pulling apart everything we're doing and going, can I make that faster? And it's not going to be 100% accurate. It's a can I make this save me enough time that I then review it, I then move on and I can just generally do things faster.
Again, I did the same demo of source to target mapping. Here is my source schema. Here's my gold schema. Generate me the SQL through the transformations. And it's okay. It's pretty pretty accurate, but not amazing. I still need to review it, but it saved me a hell of a lot of time. And as you start training it, as you start doing it, as you start giving it more information, more history, it's going to get better. Now, obviously, there's downsides.
LMS are particularly complex. People need to understand how the agent framework works. Are using autogen? What tool framework are you using to actually build it? That's currently a challenge. As you saw yesterday, it's becoming commoditized already. It's just click a button. I want a new agent and data bricks will do that behind the scenes for me. But currently, there's still a bit of technical knowledge. You still need to understand how that stuff works.
Obviously, large language models cost money. We're saying actually I'm going to develop faster, but I need to pay for this thing that's going to be doing that for me. And you need to balance that. If you only do this once, probably not worth doing all that effort and paying the cost, right? But if you're doing this over and over and over again, it's worth doing the upfront investment. The same with any kind of automation challenge.
It's not going to be accurate. You need to build in your reviews. is you need to make sure it's actually being looked at. Is it secure? You need to not send your data to third parties that you don't have a contract with. Don't just slap it into chat GPT. Actually make sure you thought about the architecture of it, which is hard because it's an internal tool we're using to accelerate stuff, but it needs that design. And yeah, there is the death of original thinking with us as a general industry thanks to AI, but that's just what we do.
So to recap, data engineering is not dead. But data engineering as we currently look at it is too slow. It is too old school. Even we're sitting there going, "Oh, yeah, but we're much better than we used to be in like the old SQL days." Actually, it has fundamentally moved on that we should be working faster. And I'm going to coin a horrible phrase. I kind of hate myself for this, but what we are turning into is data intelligence engineers.
We are moving away from traditional engineering as you know. Now you could say we're now AI engineers. You could say we're we're doing things differently. But this whole idea of going I am an AI augmented engineer. I'm still a data engineer. I'm still getting data from A to B. But if I'm not using these tools, if I'm not looking at what I do and finding opportunities to make it faster, make it just get there quicker.
In a few years, everyone else is going to be doing it quicker and I'm going to find myself behind the curve. Cool. That was my rant. Thanks everybody.

Learn more about the Databricks Data and AI platform.