Sponsored by Accenture | From Pipelines to Agents Rethinking Data Engineering for Autonomous Systems

Summary

Traditional data engineering is manual, slow, and expensive—creating bottlenecks that more headcount cannot solve, making agentic AI a structural necessity for modern data platform delivery.
An agentic data lifecycle replaces manual pipelines with five autonomous stages, and Avanade's open-source framework on Databricks demonstrates approximately 40% efficiency gains over conventional approaches.
This video demonstrates a Customer 360 use case with an ingestion agent that issues pull requests for human review, a transformation agent with data quality reasoning, and a gold table driven by ontology-based data product design.

Chapters

FAQs

Why does data engineering need to change for the agentic AI era?

Data volume, velocity, and complexity continue to grow while manual pipeline construction remains slow, expensive, and brittle—creating throughput bottlenecks that delay business insights. This video argues that agentic AI, not additional headcount, is the solution, with an autonomous lifecycle delivering approximately 40% efficiency gains.

What are the five stages of the agentic data lifecycle described in this video?

The agentic data lifecycle reimagines the full data engineering workflow as five autonomous stages where agents handle ingestion, transformation, quality validation, data product creation, and continuous optimization. The framework from Avanade is available as open source and is built on Databricks as the foundational platform.

How does the ingestion agent in the demo handle human oversight?

The ingestion agent automatically generates pull requests for new data sources and routes them through a human review workflow before merging changes into the pipeline. This human-in-the-loop design ensures that autonomy is bounded—agents act, but engineers validate before production changes are committed.

What is the role of Databricks in Avanade's agentic data engineering framework?

Databricks serves as the foundational platform for the agentic framework, providing the compute, Unity Catalog governance, and medallion architecture on which the autonomous agents operate. The framework uses Databricks' agentic orchestration capabilities to coordinate agents across ingestion, transformation, and gold table creation stages.

Full transcript

[00:08] Thank you everyone for coming to our talk. Some forward-looking statements and disclaimers. Please uh complete the survey after the talk. Give us your feedback and thank you. Thank you for coming to
[00:22] our session. uh Abdar and I will talk about data engineering reimagining in the world of agentic and automations autonomous systems. The agenda for today's talk is very very
[00:34] simple. We are going to talk about why data engineering has to change. What does the new life cycle look like in the world of agentic AI and autonomous systems and how do we reimagine this
[00:47] with data bricks? We all know where we have been when it comes to data engineering and where we have to go especially because the world
[00:59] of agentic AI affects data engineering, data platform and data platform modernizations. So what what are the drivers behind change that we need to do in uh data
[01:12] engineering? So first of all data volume uh the growth of data right we have seen over the years data continues to grow in terms of volume velocity and complexity
[01:25] right and that causes bottleneck in terms of operations. Second is that the world of data engineering is manual. Whenever you are building manual data data platform data
[01:39] pipelines it leads to weakness in the pipelines. uh it is expensive and it is slow and when the system itself is slow the business doesn't really get any uh insights out of it right one question I
[01:54] have in this audience how many of you are data engineers and how many of you are uh business side how do you how many uh what do you represent from business anybody from business here
[02:06] yeah fair representation thank you now when you have slow inside it is to bottlenecks. Earlier we used to have uh limitations in terms of compute but now
[02:18] compute is widely available to us but because of the challenges that we face we have throughput issues and efficiency right and we will show when it comes to reimagined uh data platform
[02:30] modernization using agentic AI it leads to about 40% in uh efficiency uh gains right and we'll demonstrate how that how that happens
[02:46] So the issues are clear. Manual pipelines are costly, ineffective and uh it leads to a slower delivery of insights. Right? We can solve this by using more headcounts but it will only
[03:00] delay the problem. But if you reimagine the entire life cycle of data platform and data engineering, we can see how agentic AI will help uh gain those efficiency standards. Right?
[03:13] So if you look at the standard uh process here, we know what the functional flow looks like. We know the exact process, right? It's about ingestion and integration. So you have
[03:26] disparate source systems you bring them into the platform and starting with the injection and integration right followed by transformation and preparation. So you any use cases that you might have
[03:39] either it is going to be BI, analytics or machine learning, data science, you have to prepare the data right and once the data is prepared it needs to be uh
[03:51] maintained and improved because the platform has to continuously improve for the future use cases right and if you see this process we know exactly what needs to happen and currently if you ask
[04:04] how it needs to happen the answer is manual data engineering workloads, right? Data engineers have to look at the requirements and implement those workflows step by step. You can see the complete workflow here. But if you can
[04:16] identify a system in this workflow where we know a particular task that is repeatable, deterministic and we know what the output is going to be, we can
[04:28] put agentic systems on top and then that will help us to get the efficiency in which will allow the data engineers in an elevated role of strategic governance because the agent can take care of the
[04:41] mundane task and the data engineers which are human in the loop. they will then take care of the governance and strategic insights. Right?
[04:54] So how does the new new life cycle look like when it comes to agentic AI and data platform? So if you look at the entire end to end process, we start with design. So we have to first of all look at the discovery of the entire
[05:07] operations right. So you start from discovery then leads to design, build, validate. Once you have validated your entire platform, you can then deploy the system which then needs to be maintained
[05:20] and operated at a at a scale followed by improvement. Right? So this is the new new process as to how it looks like. Now where do you see the role of automation in this? So agents at every step in this
[05:34] process can do the handoff between the subsystems which we'll demonstrate via the demo how it looks like. And then once uh the agents are incorporated into this life cycle they work with the human
[05:48] uh in tandem which will then deliver uh the efficiency gains right. So when we say agents there are agents for every other task. So for example, discovery, configuration, orchestration, ETL,
[06:00] ingestion, standardization, all those repeatable, deterministic tasks can then be passed to the AI agents, right? And then this simply brings in the benefit of the
[06:13] autonomous systems that we are going to demonstrate how the entire process can be automated minimize the errors which would have been introduced by the data engineers and uh the quality control can be
[06:25] automated on top as well. Right? So this is how the new process looks like. You can see that some activities are done by the data engineers. Some of
[06:37] the activities are completely done by the EI agents and whereas remaining activities are done jointly with the data engineers approval and validation which then be can be completed by the
[06:50] machine. Right? So here is the new approach where from all the way from discovery to deployment and operation and maintenance uh and the entire
[07:02] process is done with the humans and uh agents uh together. Now just to demonstrate how this efficiency gains comes about right. So
[07:14] if you think about your life as a data engineer if you are going to spend 40 hours uh in a week doing your data engineering task how do we go about getting those efficiencies? So if you look at the task
[07:27] that you perform in your day-to-day life we can divide them into three sections. First section is where everything in that list of activities can be automated through AI agents.
[07:40] Second block is where AI will assist your data engineers. Doesn't necessarily mean the agents will completely uh uh do the task for you but they will provide
[07:52] the assistance required for your data engineers. It will lead to some sort of uh efficiency gains. And then the third section is where the data engineer has to play significant amount of role which cannot be automated. So if you see the
[08:06] top section where about 13 hours of work is completely automated by the agents right followed by the middle section where from out of 10 hours which was done
[08:18] earlier we have about seven hours dedicated to the data engineer but three hours are done by the agent. So if you can see in total we have about 16 hours saved uh by the data engineer which is
[08:30] which is approximately 40% of gains in terms of efficiency right so now your data engineers has roughly two days in hands to do something more meaningful look at the platform maintain it and
[08:42] drive it forward for more strategic initiatives and this demonstrates S2 from end to end what we used to do earlier manual workloads and going to the automation
[08:55] systems 40% of efficiency gains can be really achieved right and we will show that in the demo that Abdur will uh do it later on as well. So when we look at the
[09:08] operationalization so the new process how does it work in terms of the architecture right we saw in the entire process there are multiple tasks so we have to build agents for each and every
[09:21] activity which can be automated and they are deterministic in nature. So discovery agent uh standardization agent so once you have your data sources identified and those data sources can
[09:34] then be ingested. So once the data is ingested, we know how the medallion architecture works. So bronze, silver and gold in the data needs to be standardized, data quality issues need to be removed and then finally a gold
[09:46] data product layer which can then be passed on for the business and end users. Right? So all these agents uh all the way from discovery, orchestration and standardization agents they will uh work in tandem alongside the super agent
[10:01] which will look at the operations and uh any bugs that happen overnight if the system is being maintained and uh everything works in an auto autonomous way. Plus the human input obviously is
[10:15] required to validate the output done by the agent as well. Right? So going going from that part how does the asset that we have developed that looks like.
[10:27] So the asset that we have developed in abanard accenture it is completely open. um whatever framework templates that we have the customers you will be able to
[10:40] see what the code behind it does and it uses all the native data bricks uh components right so from lakeflow genie lakebase and everything that we have seen uh recently announced and the
[10:54] beauty of this accelerator is that it is modular in very in nature so it's it's modularized in the sense that any component If it changes, we are able to replace it uh with the new technology
[11:07] that arrives, right? So if let's say for example if lake flow tomorrow improves, we'll be able to incorporate that those new changes fairly straightforward in this accelerator. So you have simple and
[11:19] fast deployment um ingestion, standardization, transformation, data quality checks. We all know how data quality is very important, right? And uh we can define those rules to begin with and there are
[11:31] standard rules which applies to every data source and every system which which are already done for you. So with this asset in play your teams will gain significant efficiency because then they
[11:44] don't need to do the mundane setup and uh CI/CD pipelines uh tracking of the data movement. All the metadatriven uh related uh queries will be stored for future audit purpose as well. Right.
[12:01] I'll pass it over to Abdar and he will take you will take you through the next steps of the thank you. Hi everyone. So I will walk you through how all this concepts we have seen now and the new way to do data engineering with agents
[12:13] can be implemented and reimagined within data bricks. So we will rely on existing component and data bricks and build on top of them. At the core of it, we will have the intelligent layer which is uh
[12:26] unity catalog providing lineage governance and all the context that the agents can leverage to build in an autonomous way because the agents need that context. The second part we already
[12:39] shifted within data bricks today to proactive troubleshooting and fixing leveraging genai. So genai can understand the issues analyze them and can we can build action on top of them.
[12:52] Third part will be the AI assistant. This is embedded on every step of the way on data bricks and this is probably most of you used it today using genie on every step building pipeline spark code
[13:05] and every component and also it it also benefits the users. So we have the text to SQL allowing the users to chat and interact with the data using text. So we
[13:17] see that data bricks provide multiple component each step of the way to have a fully intelligent platform. So let's take a deeper look to the intelligence
[13:29] layer which will be in the center of this. So as we know agents are good on automation. They can generate code, execute transformation, execute orchestration but they don't understand
[13:42] the business context. This is the missing piece as of today and this is where data bricks enabled new features. So here we're talking about metric views giving the semantics and business I would say business understanding around
[13:55] the different data asset we have and this morning data bricks announced ontology which will give more detail and more context to the agents allowing them to uh act in more autonomy. So an agent
[14:08] that understand the concepts behind the data, understand what is a customer, what is a revenue, how they interconnect, how they aggregate will be able to provide insight and provide
[14:20] result and outcomes that speaks to the business users and this is what builds trust. So agents will provide more trustful outcome because they understand what they're building and using ontology
[14:34] that will allow us to get a compound I would say um uh to accelerate efficiency in a compound way because the ontology will keep built up and allow us to build more
[14:48] and more component based completely on agentic AI. So how all this come together? We we saw here how data bricks provide a component
[15:00] in genie to chat with the data provide some intelligence to troubleshoot and we have component here and there but that doesn't make a data platform. A data platform goes the end toend system goes
[15:12] from data ingestion to insight with multiple steps on the way. And what we are doing here is that we're adding the agentic layer at the center of this existing component to make them smarter.
[15:25] So before going to the demo, let's take a look on the architecture we use to implement data platform using agentics. So you all recognize here the medallion
[15:39] architecture. So, Medidaling architecture our solution we built it on top of Medallion architecture which give us an understandable solution understandable data platform uh because here we are enhancing data engineering
[15:52] part and give us control and trust within the platform. So even the agent is building at a speed we understand what the agent is doing and the quality of each layer. We also will be
[16:05] leveraging existing component for data ingestion like lakeflow connect autoloadader and the agents will act as a data engineer building configuration and code on top of this architecture and
[16:19] then to deploy it from one environment to another we're leveraging data bricks asset bundles. So we have also a control and a full life cycle to deploy the data in different the the implementation in different environments.
[16:32] So this is what the architecture will look like and let's go to the demo now. So during this demo we will see a multilow agent that will ingest clean the data and then build a data product
[16:45] like customer 360 view that we will demoing today. So this is what we do every day. So on top of this platform let's go to the demo and we start with ingestion. So in ingestion it's a simple
[16:58] task. We're giving our agents requirement. So the agent has starts from a requirement explaining what tables we need to load and what frequency and what job we're grouping them. So text explaining what we want to
[17:12] do. So here we have a SQL server source adventure works. I think everyone knows adventure works. So we will use it as an example. So we have this requirement ready that business analyst or business
[17:24] user will complete over time or data engineer and then we build we are showing this through the chat UI. So I will just tell my agent hey build me the a job that is described on advant ADV
[17:38] ingestion within that specific branch. So the agent will read the requirements he knows. So he will be leveraging external tools. You have seen it's a call to the repository read that
[17:50] requirements and then prepare a plan before doing the configuration. So here my agent understood that it's a SQL server pipeline that he needs to build. He used the pattern that we associate
[18:02] with um autoload uh sorry lakeflow connect. there is a call to another tool to bring the data to create the configuration and then I have the confirmation that my agent completed the
[18:15] task. So we're doing this you're using this UI to show you all the steps how what's happening behind the scene. So here at the ingestion we are let's say at the lowest level of intelligence we
[18:27] have an agent that understand human instructions call tools and just execute the instruction we already gave him. this agent will will take this as a con uh context and then will complete the
[18:41] task and to make it I would say reliable and uh so the agent will commit the code as any developer commit the code to the ripple. So you can see here I have the
[18:53] definition of my lakeflow ingestion pipeline where I see the tables that I provided in my requirement and all the technical details. So it's in a YAML file. The agent generated it, put a
[19:06] commit, create a pull request, and then a data engineer will review the pull request to have the needed control. You know, we don't want agents to go and deploy in production, at least not right now.
[19:19] And then so we can see here that we have our pipeline that was generated. It's a lakeflow connect. Nothing I would say magical, but it accelerate the work.
[19:31] allow us to load the new tables with reliable uh pattern that we're using. Now we did the ingestion of the data. We move to the silver. So within the silver we want to do cleansing. So we need more
[19:45] from our agent. We don't want our agent just to execute instructions. We want him to think a bit more. So we will start also another workflow. We are showing uh each piece apart. So here I'm
[19:58] asking my agent to configure the table product from the schema adventure works. This is where I configured my lakeflow connect and am ask giving him the schema
[20:10] and asking my agent to go and load this uh table. So I'm calling a specific agent here which is the standardization agent. He knows that he's reading from the bronze layer and loading into the
[20:23] silver. So you can see here that we have something called agent one because it's a workflow with a multiple agents that makes the solution more reliable. We have more efficiency and lower cost
[20:35] because every agent is using a lower I would say context to achieve the goal. So first my agent validated that the table exists. So you can see it shows bronze.adventterworks.product
[20:52] and read context and guidelines of the project. So we have a shared context about the project that contain guidelines for example naming convention some rules about the PI data do not ingest P data for example sensitive data
[21:05] use snake case for naming and we can have general guidelines so we're not doing specific requirements for every table we're giving guidelines and also the agent will retrieve the metadata so
[21:19] this is what we spoke about the unity catalog being the intelligence layer. The agent can discover the metadata, the columns that we have on the tables and then based on this he will start
[21:32] reasoning and building the the cleansing rules. So we will see here that in the reasoning my agent also retrieves some data quality metrics. So he will be looking how many nulls I have in a
[21:46] specific columns, how many unique values and this is provided by lakehouse monitoring. So we are building on top of compon component of data bricks. So we have the lowest layer providing inputs
[21:59] in a controlled manner that the agent can interpret. So instead of the agent going and scanning the table and reading all the data making the context bigger and with poor result he is reading uh
[22:12] metrics that are already calculated and refreshed. After that the agent come up with requirements. So you can see here for every column what target column he decided the mapping he leveraging the
[22:24] guidelines and also the metrics and information he could discover about the data to identify what control and validation and cleansing I need to apply of each rule each column sorry. So it
[22:36] can be data cast checking if nullable is allowed. What is if my column is a key and then using this requirement my agent will trigger a pipeline. So he will test
[22:50] the configuration he come up with and you can see here that we have an execution run and we have a link sorry we have a link to ripple. So you can see we have agent number three that did the execution. It was successful. Moved to
[23:03] the next one. Created the feature branch and then we created the pull request. So we can go and take a look to this pull request that was created by the agent
[23:15] and also always in this optic to have control. So the agent generated a pull request put a comment based on the reasoning how he came to this result or she
[23:27] and then we will have the configuration of the framework because the agent is not writing code he is calling function and libraries that we already built and you can see we have the configuration in
[23:39] the repo always to control what we are putting to uh to the uh the production. We can also take a look to the execution result. So here we can see that the job executed successfully but the agents
[23:53] number four I believe he will capture the output of the pipeline. So this is when we said earlier that we have the proactive troubleshooting because using Genai we can the pipel the the workflow
[24:07] can analyze the output if there is a failure and then adapt the configuration he's doing and changing what he's developing. So he's doing mainly his unit testing.
[24:19] Now we did load the data into silver. We implemented our clean sync. We have more intelligence in our framework in our agent sorry agent workflow but we need
[24:31] more because this still just the data as I received it I cleansed it it's usable by data engineers data expert but it's not a product yet it's not a semantic
[24:43] model so in the next step we will see how we can how we can leverage AI to build the product so for this example and let Let me let the Oh yeah before
[24:57] going sorry before going there. So you can see here we have a lot of print. We made it verbose for the demo. But if we go to MLflow we can see that data bricks provide full observability and
[25:09] monitoring of the solution. We can see here that my session executed which agent was called. We can see it at the session level which agent was executed and also what external tool we called.
[25:22] All these external tools are giving us controlled execution to the plan that the agent or the LLM came up with. So we can see the whole workflow and we can
[25:34] see also the number of tokens that was consumed and the cost of this query from the LLM perspective. How much it cost me to leverage the LLM here. So with MLflow we have this full governance and
[25:47] observability and data bricks did some announcement this morning that will give us more control and more uh ability governance I would say features. Now let's go to our latest agent where I
[26:01] build my silver I want to go to my gold and I want to build a data product. So the gold layer I will just tell my agent build me a customer 360. So this is a different step of the workflow. And then
[26:14] what will happen here? My agent will leverage an ontology. So my agent will retrieve the domain where this request should belong. So he is generating a data product and the data product here.
[26:28] So he did an analysis about the domain and concluded that the the domain that he should be using is is the the sales and then he will retrieve all the entities and the description and the
[26:42] relationship between the domain entities attributes the aggregation the links between all these entities he will retrieve all the information about the um the sales domain. So here we are
[26:55] building a context to make the agent aware of the business I would say context and then the agent will generate a full requirement. So you can see here we have the aggregation rules. We have
[27:08] the list of the dimensions that we want to build and for every dimension we have requirements saying sd type two the renaming the calculation rule but also the link to the physical artifact
[27:20] feeding this component. So this is what the ontology will give us. It will give us the link between the conceptual design and the physical object to build the working product. And then from here
[27:34] our agent we will call the agent we have seen at the beginning which we call the configurator that will take this requirement. The user can review this requirements enhance them or approve
[27:46] them. And then we have the configuration agent that will generate the code out of this um out of this requirement and uh this description. So this is the last part that will give
[27:58] us an full autonomy agent building autonomously a data product and as we evolve the ontology and we maintain it the agent will be able to build more and more uh data products or component that
[28:13] are relevant to the business and make sense in the real world. So that was about the demo and I think we go to the question now. I think uh so yes so this was the our asset that we have developed
[28:26] and the beauty of this asset is that even though the new features that have been announced today in the keynote we will be incorporating them in this asset and it just makes it very easy for us to add remove or any other components right
[28:39] uh but that was the end of the our session but uh and I think we have we kept intensely 10 minutes for Q&A so just in case if you have any question answers we are happy to uh answer the questions if you have any thank

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.