The 5 Levels of Agentic Data Engineering

Summary

Datafold CEO Gleb Mezhanskiy presents a five-level maturity framework for AI in data engineering, ranging from no AI at Level 0 through tab autocomplete, agentic loops, and context engineering, up to Level 5 multi-agent teams with adversarial collaboration between lead, engineer, and reviewer agent roles.
The most important unlock for moving from chat-based generation to a true agentic loop is connecting the AI agent directly to the database so it can verify its own outputs against real data—a step many enterprise teams have not yet taken.
Data agents work most effectively as reasoning loops paired with deterministic verifiers like data diffs and unit tests, and the ultimate goal is shared internal substrates that let non-engineers inject domain context directly into the agentic workflow.

The 5 Levels of Agentic Data Engineering

As data systems scale in complexity, relying purely on manual pipelines creates an unmanageable bottleneck. Shifting from traditional hand-coded pipelines to autonomous, self-healing data workflows requires a structured framework for AI delegation.
In this video, Gleb Mezhanskiy (CEO of Datafold) introduces a foundational maturity matrix for data teams. Learn how to navigate the operational layers of automated pipeline building, testing, and error-remediation, and explore how to safely hand the keys to reasoning agents without losing data governance
𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀
• Data agents operate effectively as reasoning loops that thrive when hooked up to deterministic verifiers like data diffs and unit tests.
• Transitioning to high-level automation requires breaking multi-layered data engineering tasks into isolated modules with scoped context.
• The ultimate win-condition for data organizations relies on building shared internal substrates that let non-engineers inject direct domain context into the agentic workflow.
00:00 — Introduction to AI Productivity Gains
05:37 — Level 0: Tab Autocomplete
06:19 — Level 1: Chat-Based Generation
07:24 — Level 2: The Agentic Loop
09:32 — Level 3: Context Engineering
11:25 — Level 4: The Success Harness
13:46 — Level 5: Agent Teams
17:51 — Summary & Strategic Advice
Check out future Agentic + AI Observability events: https://luma.com/AgenticAIObservability

Chapters

FAQs

What are the five levels of agentic data engineering?

Level 0 is no AI; Level 1 is tab autocomplete tools like Copilot and Cursor; Level 2 is chat-based generation with a copy-paste bottleneck; Level 2-plus is an agentic loop where the agent connects directly to the database; Level 3 is context engineering using done files, skills, and MCP; Level 4 is a success harness with linting, tests, and data diff; Level 5 is multi-agent teams with adversarial collaboration between lead, engineer, and reviewer agent roles.

Why is connecting the agent to the database such an important step?

Connecting the agent to the database enables it to verify its own outputs against real data rather than relying purely on static context. This is the core unlock for moving from a chat-based workflow to a true agentic loop, yet many enterprise data teams have not taken this step—which this video identifies as the most common gap between teams with low and high AI productivity gains.

What is a success harness for data engineering agents?

A success harness is the Level 4 capability that combines linting, unit tests, and data diff tools to give agents deterministic feedback about whether their outputs are correct. By pairing a reasoning loop with these verifiable checks, the agent can self-correct without relying on a human to review every output produced.

How do agent teams work at Level 5?

At Level 5, multiple agents collaborate in adversarial roles: a team lead agent decomposes the task, an engineer agent implements it, and a reviewer agent critiques the output for correctness and quality. This mirrors human team structures and allows complex data engineering tasks to be broken into isolated modules with scoped context for each agent.

Full transcript

[00:02] Everyone says that they use AI in their work, but at the same time the productivity gains that everyone is reporting are vastly different. The most important unlock to get to a genetic loop in data engineering is for the agent to connect the database. Sounds
[00:14] trivial, right? But again, I'm talking to so many enterprise teams that have not done this, which blows my mind, but it's still the case.
[00:26] needs to rely on you, the less you need to iterate. Hi everyone. I'm Gleb, aka Jeff, but um
[00:38] Today we're going to talk about data engineering with AI. So, raise your hand if you are doing data engineering in your work. Okay, great. So, you came to the right
[00:50] session. Um raise your hand if you're doing software engineering in your work. Great. So, software engineering and data engineering are actually very closely related, and I think that um a lot of patterns are very broadly applicable
[01:03] across the disciplines. And what I would like to talk to today is why five levels, why framework, and
[01:15] this is not a talk about the most advanced agentic approaches. Um this is more about the mental model of how we can think about the progress in automation. So, quick intro about me. I am data engineer
[01:29] in my past life, also data PM. I got a chance to build data platforms at three different companies, Autodesk, Lyft, um a seed stage startup in autonomous vehicle space.
[01:41] And what is interesting in my career is that I got to see a lot of patterns in terms of what's working and what's not, and that was all pre-AI. And then I started Datafold learning from my experience as data engineer with
[01:53] a mission to automate data engineering as much as possible. And again, that was pre-AI, so before LLMs were there, we were really focused on trying to decompose the data engineering workflow, take some of the most tedious workflows
[02:05] like QA, for example, change management, and then build tools to automate that. And now with AI, of course, we can do much more. So, um our work and life became way more exciting.
[02:18] And the reason why I thought that talking about the levels of AI in data engineering is interesting is because when I talk to data practitioners, everyone says that they use AI in their
[02:30] work. But at the same time, the productivity gains that everyone is reporting are vastly different. Actually, for this talk, I was like, maybe I should put some charts on since I'm a data guy, so it's supposed to be data-driven. But if
[02:43] you look at what people are reporting, it's widely different, you know, jobs are lost, jobs are gained, companies are struggling to hire, people are struggling to find jobs. It's all over the place. Question to you, so who feels
[02:55] that AI made you at least twice as productive over, let's say, a year and a half ago. Awesome. Who feels that they are 10x more productive? All right.
[03:07] Who feels that not much difference, not sure? Okay. So, this is a pretty advanced crowd.
[03:20] I think that's the really interesting question, especially if you are, let's say, very advanced in the use of AI, but maybe your peers aren't, or maybe your customers aren't, or maybe you are going
[03:33] into a place that feels like they are so far behind, is how to reason about progress in AI adoption. And I've been thinking about what is the differentiator, what is that makes some
[03:45] teams and data practitioners feel super productive, like 10x, right? Or some feel like, yeah, it's it's helpful, but it's not really a deal breaker. Is it models? Some using more advanced
[03:58] models? Is that CLI? Is it like using cloud code versus Codex? Is it prompting skills? And I think ultimately the question to ask is
[04:10] very simple. How much can AI do for us for a given task? What is the level of automation versus how much actual manual work we have to apply, whether
[04:22] that's prompting, re-prompting, typing in context, or otherwise copy-pasting things, or in general interacting, right? Or doing the rest of the workflow manually. And
[04:35] if we can pose this question, then maybe we can think about some levels of to what degree our workflow is automated. And that definitely applies to software engineering. I think it applies to a lot of roles, and I think it applies to data
[04:48] engineering. And what I want to do today with you is to walk through level zero to level five, and think about what they are. And the reason why I think it's interesting is because, again,
[05:02] if you want to get more productive, it's helpful to know what's out there. If you're trying to level up your team, which is important because if you're very productive, but your team is, let's say, on less advanced levels, then they are
[05:14] slowing you down because you may be getting out pull requests really fast, but if it takes everyone to review it, let's say, 5 days because they're just reading every single line of code, everyone gets slowed down, right? Or if
[05:26] you're talking or working with um very large companies that have very wide adoption cycles, how do you even reason about the road map? So, level zero,
[05:38] tab auto complete, right? Super obvious. Who hasn't tried it? Copilot, Cursor, I think, is a tool that got tab auto complete really popular. And it's really a no-brainer to use
[05:50] today. The reason why I it's I'm talking about it here at an event which is supposedly about advanced AI is because I'm still talking to a lot of teams and when I ask them, "Hey, like how do you use AI?"
[06:03] They say, "Well, it's actually really great at auto complete. I'm like really finding great." And then that's pretty much how they use it. That's where the usage caps out. But it's already a start, right? Because when someone starts to feel that, "Okay,
[06:15] it can actually guess the next step for me. It can already make me more productive." That's a good start. Now, level one is using the chat. Surprisingly, especially enterprise, when I talk to practitioners and ask them, "Okay, what is your AI workflow?
[06:28] How AI is helping you?" They're telling me something like, "I am going to chat GPT or cloud, whatever is that approved AI at the company, and I'm asking it to write me a SQL query,
[06:41] and that's very helpful. And then they take that SQL query and they paste it into Snowflake and they run it. And then depending on the result, they paste the result back and give the feedback, right? Now, that's something.
[06:54] The problem here is that data engineer is the bottleneck. The human is the bottleneck. And there are variations of that. I also talked to teams who are saying, "Well, we're using cursor." And it's the same thing. They're basically asking the agent in the cursor to write them code, then they take it
[07:07] and run the database. The problem here is that, again, you're the bottleneck and so your job is copy-pasting and piping LLM output into different tools. But again, that gets you into at least leveraging AI to not
[07:22] to automate the code writing. Now, the level two is where it gets very interesting. And I think that's where it starts really clicking for teams and going from, "Okay, maybe I'm getting like 20% more productive to I'm getting
[07:34] at least twice as productive and more." And the agentic loop is a very simple idea, right? So, an agent is a system that is able to work towards a goal using tools, applying those tools, and deciding how to use those tools, and
[07:47] iterating until it accomplishes the goal that you give it to. And so again, in non-agentic workflow like copy-pasting into chats or agents or etc., you're the bottleneck, but in the agentic loop, you can
[08:00] set the task, and then the agent thinks about the task, plans it, and executes something, uses tools. And again, we're talking about data engineering. So, the most important unlock to get agentic loop in data
[08:13] engineering is for the agent to connect to database. Sounds trivial, right? But again, I'm talking to so many enterprise teams that have not done this, which blows my mind, but it's still the case. And not only they haven't done this, but
[08:25] they haven't even felt the urgency of doing this because they haven't felt what the agentic loop actually gives you in terms of impact. And then if you start talking to them, well, this is so important and so impactful, the biggest pushback you get
[08:37] was, well, it's data security, and we can't expose LLMs to our data. Now, that's a very reasonable pushback, but right now, it's actually possible to run agents in a fully secure
[08:51] environment. By that, I mean that the biggest concern in general enterprise is that data cannot leave your security perimeter, it cannot get exposed to LLMs outside of your security perimeter. But most modern data platforms, including Databricks and Snowflake and
[09:05] Google, now have their own coding agents and LLMs that you can run within your data environment. For example, if you're using Snowflake Cortex Code or Databricks Genie Code,
[09:17] those calls to LLMs and calls to database happen within the same security perimeter, so the security should be happy. But again, this is a really, really important step because going from top complete and chats to agentic loop
[09:29] really is where you start to see the productivity gains. Level three is context engineering. Well, the context engineering has been a really big topic in 2025. I think we're still barely scratching it, and it's not
[09:42] like a level that just you start doing this and then you're done. I think that it spans everything you do once you start investing into it. And everything for an LLM is context, right?
[09:55] And it spans agents and D files and skills and any documentation you reference and MCP calls that help LLM figure out. And it's really, really broad, but
[10:07] ultimately comes down to telling the agent exactly the information that it needs to accomplish the tasks. Less is more. And then the more you invest in curating the context, obviously, the less the agent needs to
[10:20] rely on you, the less you need to iterate. And speaking about teams, context engineering is what really makes the superpowers shareable. Because if I am really good at my workflow, but my
[10:33] team is really lagging me behind, helping them onboard through better context, better documentation, better agents and D, is really what levels up the entire team. And again, this level is not
[10:46] really like you step over it. This is something that you actually carry forward across all the other other levels. For example, I'm not doing as much data engineering anymore in my role since I'm running a
[10:59] company as the CEO, but I have a data vault CEO skill, which actually is in my done file that contains information about the company, all the people, their roles, what are we doing, who our partners are, how our sales
[11:12] motion works, how our contracting works. And so, most of the tasks in my day-to-day workflow, I run through the skill because it just gives the agent immediate understanding of who I am and what I'm trying to accomplish. And the same can be done in
[11:24] any role. Now, level four is the success harness. Now, if we go back to this agentic loop idea, it looks really good, right? Agent just autonomously creates database, potentially makes MCP calls, CLI tools.
[11:39] Everything is great, except who says what the success looks like? In level two, the agent kind of implicitly decides what is the success. And for example, if I'm asking the agent
[11:52] to produce analysis, it thinks it succeeded when it created a SQL query, run it, the query run successfully, and then it presents me the result, right? But who says whether the result is correct? No one defined
[12:04] it. Or if I want to build a DBT model, the agent probably would assume that the success is when the DBT model successfully, well, runs without errors, right? But you can really automate significant
[12:18] part of your workflow unless you define the success criteria. And this is where the success harness comes into play. The idea here is very simple. You want to define a deterministic
[12:31] success for the agent when you delegate a task. And by deterministic, I mean that it's very clear and it's not subject to LLM inference or smartness. It's very clear pass or fail criteria.
[12:45] Examples are linting, right? So, linter either succeeds or fails, and if it fails, it tells you what's wrong. Query runs or fails. Test suite, you execute it, and then some test fails, and then the the model
[12:57] knows the agent knows that um it has to fix those. Another one specifically for data engineering is data diff. So, data diff is actually comparing data sets and understanding how the data diverges. That's something that we use a lot at Datafold for um
[13:11] for example, data platform migrations, cuz you want the the the data parity. So, success current harness can come in different forms, but it's really critical to be able to get more autonomy out of the agent if you
[13:24] are able to have a very objective success criteria for whatever task you're delegating. Because then you can run these agents in background, you can run them in CI, you can run them without actually having to babysit them because
[13:37] you know that if it doesn't succeed, it will try again. And if it succeeded, then you can trust the output. Now, level five is where it gets really, really interesting. So, the big problem with
[13:51] um code harnesses in general, even with the state-of-the-art ones like Codex and Cloud Code, is that they are good, at least right now, at accomplishing pretty impressive tasks,
[14:04] but they still are quite challenge challenged by large-scope projects. For example, I haven't told you what Datafold actually does for for living. We are a made data engineering, and specifically,
[14:17] we take large-scale enterprise projects, for example, a data platform migration, like moving from Oracle to Databricks, and we provide that as a service. So, essentially, we say we will migrate you
[14:29] the customer from Oracle to Databricks, we write millions of lines of code, and prove that all the data is the same. Something that previously you would hire a services company, a SI, a consultant for. Migration is a very large-scale project.
[14:43] It involves rewriting millions of lines of code that produce at enterprise scale thousands of assets. And so, managing such a project, you cannot really do with a CLI harness, no matter how smart it is. So, that's why
[14:55] where the idea of agent teams come from, where you can take a very large-scope project, you can slice it into tasks, and then have teams of agents solve those tasks
[15:07] successfully, and then every all the result coming back to you as a user. And this is really where the frontier is right now. So, for example, Cloud Code has the agent team mode, which is experimental, you can enable it with uh
[15:19] an environmental variable. And this is roughly how the architecture looks like. The idea is that you have different teams, teammates that are working on specific tasks and drawing from tasks task lists and kind of pushing back the
[15:32] updates. And they also communicate with each other, which is very interesting idea. Um at Datafold we use a little bit different structure. We are have um separate pods that do not communicate with each other. They all go through the
[15:44] team lead. And we also have different roles for agents. For example, for migration, we have engineers who are writing code and we have reviewers who are reviewing the code. And they all have separate roles and separate context.
[15:56] One interesting thing here is that if you start dividing roles between the agents, you can actually get much better result than having a single agent accomplish a task because you can take advantage of kind of adversarial
[16:10] adversarial collaboration as I name it. For example, if you have an engineer who is responsible for both doing the work and doing the validation, you will quickly discover that it will try all the hoops and it will try to
[16:22] convince you and itself that it succeeded even though it didn't succeed. So, for example, you will tell it to execute test suite, it will execute a test suite, some tests will fail, and it will try to explain elaboratively, for example, that the
[16:35] tests that failed actually don't matter or those tests are completely wrong tests. But if you separate those roles and you make the QA more of adversarial process where the engineer is supposed to produce the work and the reviewer is
[16:47] tasked with actually doing a very comprehensive QA, then you have a better chance of success because your reviewer really tries to optimize for finding issues without
[16:59] having to like the definition of success for a viewer is very different for for the engineer. So, that's one trick that you can use with agent teams. And then the another problem that has to be solved here is how do you manage context across all the agents? So, for example,
[17:11] in Claude they take advantage of markdown files extensively, which is a very simple and elegant idea. Um for us, we use a knowledge graph. We actually use a graph database that tracks the entire project and all the
[17:23] relationships with it. It's a bit more complex approach, but for what we're trying to do, like a data platform migration, where you have very complex relationships between all the data sets, we found it working much better. Agent teams is level five. So, the question
[17:36] is, do we jump straight to it, right? Whatever whatever the level is of your team or your company. I think that to get there, you probably want to climb through the levels and make sure that everyone is comfortable
[17:49] with all the previous ones. Because if you um want to use agent teams, there are really good reasons to do that. You can take on large projects, for example, a massive code refactoring or creating a large application.
[18:01] And you can take advantage of specialized agents, and they can you can do adversarial collaboration. It's all great. You can lead to higher data quality. But you also take on more complexity because now you have a lots
[18:13] of different things that you need to optimize. You have to optimize the context for each agent, each role. It's also much higher cost. So, much easier to burn through tokens.
[18:25] I really recommend mastering the foundational levels first because then you can kind of build the building blocks that will then allow you to make uh take advantage of the more advanced features. So, again,
[18:38] five levels, very straightforward progression. And what I would do is I would think about where your team is at right now. Where are you right now? How to go
[18:50] forward, how to level up, what are the things that are missing, what are the things you could be doing to um for example, take ideas from level three, like context engineering, and apply them in the higher levels. And
[19:03] I think that the big disparity in productivity can really be solved if we have a shared common language of of understanding of AI. For example, if you have executives that are
[19:15] saying "We should be more productive and we should be investing more in AI." It's really hard to actually create a roadmap without having a shared language and shared understanding of actually what does more look like? Is it maxing
[19:28] tokens? And just you know, spinning up 10 10 sessions of of your favorite agents. Or is it trying to get to better automation through
[19:40] things like that, right? Different patterns that actually get you um to higher degree of automation. So, that's what I want to leave you with today.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.