Agentic Data Engineering on Spark: Why AI Needs Context, Control and Validation

Summary

• Full context: essential for agentic data engineering in production across data, pipelines and infrastructure.
• Context and validation: prevent SLA breaks where LLM-only optimization ignores runtime constraints.
• Runtime and cost savings: in-driver context cut runtime 65% and costs 90% in a Spark optimization test.

Agentic Data Engineering on Spark: Why AI Needs Context, Control and Validation

Talk by Roy Daniel, Co-Founder & CEO, definity
Agentic data engineering works in production only when agents have three things: full context across data, pipelines and infrastructure, control to act at runtime, and automated validation before changes ship. This video tests an LLM connected to Databricks, Unity Catalog and the Spark History Server on a real Spark job, where the LLM's recommendation cut cost about 40% but stretched runtime to 1 hour 7 minutes and broke the hourly SLA, while in-driver context traced the true bottleneck to a single line of code and cut runtime from 55 to 20 minutes at 90% lower cost.
In the test, an LLM recommendation cut a Spark job's cost about 40% but stretched the runtime to 1 hour 7 minutes, breaking the hourly SLA. With in-driver context, including S3 listing operations, definity traced the problem to a specific line of code, lowering peak vCores to 16, cutting runtime by 65% and cost by over 90%. Roy Daniel also demos runtime prevention of data freshness incidents, root-cause analysis with lineage, one-push auto-tuning, and customer results including a 44% Spark platform optimization and 9x ROI in five months.
Sponsored by definity.

Chapters

• 00:00 Introduction
• 00:31 Why context matters: the limits of data observability
• 04:10 Testing an LLM to optimize a Spark job
• 05:39 LLM-only optimization: cheaper, slower, SLA broken
• 07:09 Three primitives: full context, runtime control, validation
• 08:23 Agentic optimization result: 55 to 20 minutes, 90% lower cost
• 11:30 What is definity: an in-driver monitoring agent for Spark
• 14:01 Demo: lineage, freshness incidents and runtime prevention
• 16:56 Customer results: 44% Spark optimization and 9x ROI

FAQs

How do AI agents optimize Spark jobs?

AI agents can reduce costs, but they often ignore runtime constraints and SLAs. This video shows why full context across data pipelines and infrastructure is essential—in-driver context traces actual bottlenecks to identify the true problem rather than just recommending resource changes.

What does Databricks Data and AI platform need for AI agents?

Agentic data engineering requires three things: full context across all data, pipelines and infrastructure; control to make changes at runtime; and automated validation before shipping changes.

Can AI cut Spark job costs?

Yes, but without proper context it can break SLAs. This video demonstrates how in-driver context cut runtime from 55 to 20 minutes (65% faster) while reducing costs by over 90%, proving that context matters more than raw recommendations.

What are production patterns for agents on Spark?

The video covers runtime prevention of data freshness incidents, root-cause analysis with lineage, one-push auto-tuning, and shows customer results including 44% Spark platform optimization and 9x ROI within five months.

Full transcript

00:00 Introduction

Hello everyone. Welcome. I hope you all have a great first day at Data Bake Summit. My name is Roy. I am the CEO and co-founder of Accelity. Today we'll talk about agent-based data engineering and what it actually takes to implement it in production. About the three fundamental foundations needed to do this, and how Accelity fits into it all . Well, there was just a goal in the World Cup, so everything is good. Let's begin.
So, I think the key word today is context. We all heard this earlier today during the keynote. So let's start with context. Over the past few years, data observability has been a great first step in giving data engineers and data platform teams insight into what is actually happening with their data stack. And to really get that level of visibility, we need a complete view of the stack: from how the data behaves, what its quality is, what the schema looks like, to pipeline operations, failures, successes, retries, SLAs; to the actual execution, from code through the environment to the origin and how data moves, and the pipelines generate, read, and process that data; down to infrastructure performance and costs.

00:31 Why context matters: the limits of data observability

There are now many solutions that have attempted to solve this very complex problem. The problem was that they were addressing very specific and narrow levels of the stack: either they were only looking at the raw data at the “gold” level and mostly at data quality, without considering the entire process behind it, or they were looking at infrastructure, performance, and costs at the machine level. For example, understanding what my costs are or how many machines I run, but without understanding what workloads are being performed on them.
So with this external view, which is usually applied after the fact, there was a lack of context and actionability, which prevented data engineers from optimizing their platform, performance, preventing incidents, finding their root causes, remediating or validating any changes they made to their pipelines. So, for example, if we had any issues or notifications: "Hey, my data is out of date." But they are already at my "gold" level.
So what do I do with this and where did it come from? These are questions that were very, very difficult to answer. If I have a task that meets its SLA, why is this happening? Where does this come from? What is the performance bottleneck? Again, these are questions that were very, very difficult to answer. And, of course, everyone is concerned about costs. So it's great to know that my expenses skyrocketed last week, but how do I optimize them?
What exactly is the inefficiency? Again, without a deep contextual understanding, this was very, very difficult to answer. So, we're moving towards 2026, and we all want to leverage agent capabilities on top of our monitoring and have full context in the platform. The first thing we want to do is maybe connect my model to it . And it doesn't matter if it's the cloud, or Gemini, or something else, like OpenAI. We will try to connect it to Databricks, Unity Catalog, maybe Spark History Server or my GitHub.
And then when I ask how to optimize this task, I might get a somewhat general answer. And the main questions that arise when we try to implement this into production: is the result relevant to me? For my work, for how my data and infrastructure behave? Or ... am I just getting an offer? Or can my agent operate in a production environment and make efficient and reliable changes? Can I entrust him with production work?

04:10 Testing an LLM to optimize a Spark job

So we decided to test it in practice. We took a Spark job, and this is how it looks in the Spark UI. Sure, we all love Spark UI, but it's not the best visualization imaginable. Let's take a moment to see what this actually looks like— this is the Definity representation, this is what a Spark job looks like. There are three previous queries, they require 100 virtual cores to execute, and it takes almost an hour. As you can see, the first problem is that the load on computing resources is very low.
Again, this look is great in Definity , it's not something that's easy to see in Spark UI. Now let's transfer all this context from Spark History Server directly to the cloud. And the cloud does a great job of analyzing this particular task . It provides recommendations on how we can optimize it. You can see that the number of worker nodes is specified here, as well as the minimum and maximum parameters that we need.
So what does this look like in real production? If we implement this recommendation, the result will look exactly like this. And, as you can see, first of all, he did a pretty good job of optimizing the task. So, we have already reduced the number of virtual cores from 100 to 50-80. Ahem. But the big problem is that we actually increased the runtime to an hour and 7 minutes. So, overall, we optimized costs from 75,000 per year to 45,000, about 40%, by reducing computation, but we increased execution time.

05:39 LLM-only optimization: cheaper, slower, SLA broken

So, overall, it looks good, but the cloud solution lacked full context. It relied only on Spark History Server and Git. It actually missed the main reason for this inefficiency, which we'll show in a few seconds. He had no real control, so we needed an engineer to implement the recommendation, and we lost the full feedback loop. So while a 40% optimization is great, it actually lost a lot of value, which we'll show you again in 3 seconds.
And most importantly, we did n't have proper vetting. So by implementing this type of recommendation, we actually violated the pipeline's hourly SLA . This is the first thing we want to do when we have pipelines running in production, to make sure we don't break the performance with code or configuration changes. So, why does this happen? First, we said that the cloud lacks context. To have a truly proper agent-based data engineering workflow , you need the full context: from the data through the pipeline and execution all the way to the cost of the infrastructure over time.

07:09 Three primitives: full context, runtime control, validation

Secondly, he lacks control. We need to be able to actually take action during execution within the pipeline. We need to be able to have a real feedback loop to save engineers time and have automated validation—that's the third element. We must have confidence and safeguards built directly into the workflow. Because recommendations are unreliable. Sometimes we'll get unexpected results, and we want to make sure we can run them in a proper environment first , test them, and make sure both the data and the performance are reliable when we implement recommendations or any optimizations, code changes, or even fixes, right?
This is important for the data pipeline . So let's see what it looks like when you properly do agent-based data engineering with these three primitives as the foundation for what you're going to do. And of course, today we'll do it together with Definitive. So, first of all, I repeat, this was our actual pipeline launch, the original pipeline launch. We see here that Definitive, first, having the full context, correlation data, pipeline, and infrastructure, actually detected a slow inbound event or inbound load before the pipeline started.

08:23 Agentic optimization result: 55 to 20 minutes, 90% lower cost

Definitive not only profiles the behavior of the actual execution and processing, but also profiles the behavior of the driver that allows us to do this. So, as you can see here, another full context metric that Definitive has that Spark History Server doesn't have is S3 listing operations. By looking at all of this together, Definitive can provide complete, holistic recommendations for optimizing the performance of this task.
First, again, it gives an understanding of what the main problem is. And secondly, it indicates the specific line in the code where the problem occurs and what needs to be changed. In this case, the problem was related to section truncation and an excess listing event that needed to be changed. So, this is what the task looks like after Definitive optimizes it. Now the scale here is a little misleading. First of all, it may look at the same level, but now we have reduced it to a maximum of 16 virtual cores.
Remember that previous optimizations took 50 to 80. Second, we reduced the execution time from the original 55 minutes and hour and 7 minutes with the previous recommendation to 19, say 20 minutes. And thirdly, we have significantly reduced costs. So, overall, we reduced lead time by 65% ​​and reduced costs by over 90%. And then, on the reliability side, think about how much more proactive the data engineer who owns that pipeline becomes when they have a pipeline that's supposed to complete every hour, and now it completes in 20 minutes instead of 55.
If there's a problem, if there's a retry, they now have a lot more time to be proactive and respond. So, with Definitive, we actually got an accurate, efficient, and automated setup that was validated and that led to the desired result. She had the full context to actually identify and address the underlying problem. We actually have the control and feedback loop to ensure 90% cost optimization. And we have automated validation to ensure there is no degradation in performance, cost, or data reliability.
So, what is Definitive? Definitive is an intelligent data platform for the Spark and Lakehouse ecosystems. It is based on a very complex agent, it is a monitoring agent, not an AI agent, but a monitoring agent that works internally, together with the pipeline driver and directly in the executors themselves. It runs in the driver, again, not at the platform level, so it does n't matter if it's Databricks or Spark on Kubernetes.

11:30 What is definity: an in-driver monitoring agent for Spark

Because of this, it has several very powerful characteristics. First, it sees everything: from the data, every read and write, through the pipeline and execution, dynamically inferring its own linearity, e. g. environment, code, all the way to infrastructure and costs, natively, at runtime. It doesn't rely on your orchestrator, platform, or cloud provider, any of that. Secondly, he actually has the ability to control the conveyor.
So he doesn't just listen, he actually leads. Based on this, it provides intelligence at runtime. So, the definitive course provides these three fundamental primitives that are necessary for data engineering, hygienic data engineering. It provides full context, which can of course be consumed through our NCP. It provides actual control of the pipeline at runtime and provides automatic validation so we can test any change we require.
In addition to all this, we provide identical workflow agents that perform cost optimization, real-time incident prevention, incident resolution , code review, and more. Why should this concern you? Because ultimately, this will allow you to not only use these base components and potentially build your own agents, but also use ready-made agents that will help you optimize platform performance and cost, prevent incidents in real time, resolve, troubleshoot , resolve incidents 10x faster, and validate any code changes, configuration, platform upgrades, or migrations out of the box.
So let's see what it actually looks like. Here we see the full context that Affinity tracks “out of the box ”, the deep linearity between the pipeline and the datasets we generate dynamically, and the deep context we have for each job at the query level, codebase level, linearity within the job, all the way down to hundreds of different metrics from data quality, schema freshness, distribution, pipeline health, SLAs, all the way down to infrastructure performance.

14:01 Demo: lineage, freshness incidents and runtime prevention

And now we will see how it works in practice . So, this is a data freshness incident that we detect at runtime before the pipeline even starts. Immediately after installation, without any rule settings, we detect this anomaly, and with the help of our AI assistant, we clearly see its source. The AI system establishes the root cause directly in the output stream, and now we can determine where it comes from in two clicks, since we have full end-to-end data linearity.
We see that the table that was supposed to be created by the output pipeline did not appear, and we see that this pipeline had a problem with data distribution. As we mentioned earlier, Definity has the ability to prevent problems from spreading downstream. In this case, our agent actually stopped the real-time execution to avoid contaminating your "gold" tier and downstream consumers with this incorrect data. Here we see how Definity immediately monitors all costs and performance, identifying losses and providing specific recommendations for optimization.
The best part is that with the push of a button you can automatically configure the system. So, from now on, Definitive will automatically configure this task to the optimal parameters , which will again help save money or improve productivity. Here's another example of how Definitive detects a problem with KNN by essentially digging deep into the code and how it affects the data. We see here a 10,000-line logical plan and how Definitive very quickly identifies a specific line of code where a huge amount of filtering is taking place , making your task extremely inefficient.
Running on huge amounts of data when you can actually prevent this filtering. And lastly : here we see an auto-check mechanism that allows you to check any changes in the code or environment configuration and drill down into the specific line of code that is causing issues with data reliability, pipeline performance, or runtime overhead, or of course, fix it until everything is done correctly. So, to give a few examples of how this works in practice.

16:56 Customer results: 44% Spark optimization and 9x ROI

We work with a global EdTech company running Spark on Kubernetes and helped them optimize their Spark platform by 44%. By 44% their Spark platform and increase data development speed by 70%. The most enjoyable part is that they received a 9x return on investment in just the first 5 months, which shows how quickly these agent capabilities are being realized. Secondly, I want to talk about one of the leading AI platforms of the digital generation.
You're probably all using it today to write some of your texts. We help them with similar automatic optimization in the Databricks environment, completely improving their data engineering work. This allows them to focus on what is really important to the business, instead of background engineering work that takes a lot of time and effort , requiring expert knowledge, even though they just want to move to a much more agent-based and automated approach.
And I think Eric's quote here speaks for itself . So, if you want to learn more or see a demo, we'd love to see you all at our booth 631. I'll be there for the rest of the week. You will probably see all of our " human agents" there, we will be happy to chat and answer questions. Thank you very much .

Learn more about the Databricks Data and AI platform.