Sponsored: Cognizant The Autonomous Data Value Chain How Agent Bricks is Redefining Data Engineering

Summary

Cognizant's Agentic AI-driven Data Value Chain replaces manual, pod-dependent data engineering workflows with domain-specific Agent Bricks on the Databricks Data and AI platform, organized into coordinated multi-agent hubs for discovery, modeling, ingestion, quality, governance, and operations.
Rather than sequential handoffs between specialized pods, the architecture enables human AI operators to direct multiple agent hubs that collaborate through orchestration, shifting engineers from writing glue code to overseeing intelligent agents that continuously optimize.
The approach targets accelerated time to market and productivity improvements by automating source-to-target mapping, semantic layer building, and data modeling steps that previously created bottlenecks between enterprise data teams.

Chapters

FAQs

What is the data value chain and why is it still manual in most enterprises?

The data value chain covers the full lifecycle of enterprise data from discovery and modeling through ingestion, quality checks, governance, and operations. Most organizations still manage this manually because each phase is handled by separate specialized pods that create sequential dependencies — no team can progress without waiting for the outputs of the previous one.

How does Agent Bricks serve as a production foundation for the data value chain on Databricks?

Agent Bricks provides a production-grade foundation for building intelligent, cost-aware agents on the Databricks Data and AI platform. In Cognizant's architecture, each phase of the data value chain is handled by a dedicated agent hub built on Agent Bricks, with multiple agentic solutions within each hub collaborating to complete tasks that previously required manual handoffs between teams.

What does the agentified pod structure look like in practice?

In the agentified structure described in this video, humans in each pod become AI operators who direct agent hubs rather than performing manual data tasks themselves. The same medallion architecture and data products are produced, but discovery, analysis, and modeling work is handled by multi-agent AI solutions, removing the sequential dependency bottlenecks between teams.

What business outcomes does Cognizant target with this approach?

Cognizant's approach targets accelerated time to market, improved productivity, cost optimization, and AI governance compliance for enterprise data teams. The architecture is designed for organizations that need to deliver data value faster without dismantling their existing lakehouse foundation or their existing organizational structure.

Full transcript

[00:09] I'm a principal architect with Cognizant and this year I'm also a Databricks AI MVP. And Databricks made me wear this, they they wanted me to brag about it, so uh ignore that. And uh today I'm going to show you
[00:21] uh how we can, without tying the underlying Databricks lakehouse, how can we agentify our solutions? So that is what we are going to talk about.
[00:34] So today we'll be discussing about how what is the best practices on creating agents, accelerating the time to market, and optimizing the productivity improvement and cost optimization, and
[00:47] the overall governance as- aspect of it, the AI governance of aspect of it. So we have worked with a lot of enterprises and this is uh the current
[01:00] state of most of the enterprises that we work with. So what happens is there are a lot of pods that are working. So typically a medallion architecture is created, there are like
[01:13] customer experience uh their supply chain, different lines of business across uh working as a data products, and then there are pods which are product teams, the frameworks team, and uh these they build all sort of
[01:27] frameworks, the data ingestion framework, the data ETL framework, uh the data quality validation framework, so many frameworks. What is this is a perfect example of how
[01:39] there is dependency on that we are creating between each of these pods. None of these pods can work together unless they get the inputs from the previous pod. And this dependency is
[01:51] what we have created. And now if we are agentifying these pod structure, how do we do that effectively and how do we do that in a way that is going to help business achieve the
[02:04] value in a short span of time. That is what we are going to talk about. So, it is the same thing, exact same construct where we are having the same
[02:16] data products that we are creating. We are creating the same medallion architecture in the lake lakehouse. But here, we are having the humans in the pod work as AI operators. They are operating the agent
[02:30] hub and there are multiple agent hubs here. So, each of these EDR the agent hubs like there is an So, typically like we have a requirement document requirement that we get from
[02:42] the business team and then we get the do the discovery, analysis and then the EDA, data modeling, data profiling across the entire data value chain. We are having multiple multi-agent AI
[02:55] solutions. So, that for the discovery and analysis, we have an agent hub which has multiple agentic solutions. Each of them are multi-agentic solutions. So, there are multiple agents created. And
[03:07] these multi-agentic AI solution is being operated by the pod team. So, they are not like waiting for anyone now. They will be operating the AI solutions which will help them achieve the and create
[03:20] the data products that are needed. And that that is what we are This is a representative version of that. And now to the functional view. So, the I I don't want to go inside the and every
[03:34] box on this one, but I will walk you through the top row. So, for example, there is data discovery and analysis hub. So, here we will have a separate agentic solution for the
[03:46] discovery, separate agentic solution for the modeling, separate one for the data engineering and post which there is a data quality data validation hub and then there is a for the governance and
[03:59] then there is X ops. It could be platform ops, it could be AI ops, ML ops, all these ops operations as well. And now this hub is a team of specialist
[04:11] who will not block anyone. So previously when there were humans, the block was there which was causing a dependency which was causing a delay. And that delay and dependency is now removed by
[04:23] bringing in these agentic hubs. And these agentic hubs are going to get the response in a very short span of time removing the dependency and the blocker. And that is the main advantage that we
[04:35] are bringing in here. And moving on, so this is a typical view of how the entire life cycle is going to be. So now
[04:47] when we start off a data product owner, he's going to give a requirement document and that requirement document is typically not a clear document. There are going to be multiple iterations and
[04:59] all those things. Here when we are getting it that that first DRD the data requirement document V1 is received that triggers the entire delivery. Previously
[05:12] how it was like when when it comes again like there are so many humans in the loop like they goes back and forth the the back and forth conversation is avoided here and this acts as a trigger
[05:24] and this trigger is essentially going to trigger the supervisor agent and the supervisor agent is going to call in we can again tune this. So let's say like I just want to do the data discovery. I
[05:36] don't want don't want to do anything else. I just want to do the discovery analysis and I'm not going to do the engineering. We can again tweak this. So but I will walk you through the entire uh
[05:48] value chain. So, it goes to the data delivery supervisor. And that supervisor is going to call the data discovery as the first agent hub. And that inside that agent hub, we are starting off with the data
[06:01] discovery agent, data asset analyzer, and the data profiling agent. And this is just the beginning. In the same agent hub, we plant we can have more and more more agents as there is requirement uh
[06:14] as there is a requirement for it. And each of these agents are multi-agent AI solution. And once the discovery is completed, it moves on to the data delivery. And there it starts off with
[06:26] the data ingestion. Again, uh the data ingestion since uh we are in Databricks, it can be a simple autoloader uh calling an autoloader, it can be an event-triggered option. It can be across all these uh
[06:39] Databricks. Now, we have like all the uh Lakehouse designer for bringing in data. Uh we have the agentic Lakehouse designer for doing the ETL. We can bring in that as well. That was just released
[06:52] today, so it was not called out here, but we can bring in those. And then once the data engineering is completed, then the data uh uh ingestion is completed, we move on to the data modeling where we have the logical data modeling as a separate
[07:06] agent. And then we have the physical data modeling as the next agent. And then we have the source-to-target modeling. Uh so source-to-target mapping. So, the source-to-target mapping, which is always a challenge,
[07:18] which even Ali mentioned today, one fine day the source changes. And then we shouldn't be like uh at 2:00 a.m. we shouldn't be looking at the source schema and figure out what is wrong. We need to figure out the agent needs to
[07:30] work on that. And we have a multi-agent AI solution for that. And then it moves on to the data transformation, wherein we use the Lakehouse designer, and which will help in creating the medallion architecture in
[07:43] the data lake, which is the V3 of the DRD. And underlying, we will also have the feature engineering hub, for which we can use the Databricks feature store, and which will capture all the features
[07:55] and signals for the particular line of business. And this is that this entire data delivery, it's completely event-driven, and all we need to do is just push the data requirement document V1 there, and
[08:08] everything is triggered in a step-by-step manner. And now we are our data is like uh transformed, engineered, and loaded to the medallion architecture in the bronze, silver, and gold layer.
[08:20] And the feature engineering is also done, which can be leveraged for the um ML team and the uh BI team as well. And then, for the BI team, we are also leveraging the
[08:32] semantic layer, so we can we are we can bring the build the semantic view builder, which can create the metrics view. And this metrics view can be used by the GenBI application. So, that is
[08:44] where the data apps and visualization hub comes. And in this hub, we can create the GenBI, and we can also create the data the apps as well. So, we can leverage the Databricks apps for
[08:56] building the application. All we need to do is like give it prompt, and and that prompt will be taken from the initial requirement document. So, you get the reports, and you the reports are there
[09:08] in the Databricks dashboard, and the app is built in Databricks apps. So, then the apps and visualization is also done. Now, how do we collaborate with the other teams? So, we we have the reports, we have the apps, but there are multiple
[09:22] lines of business in within the enterprise. They will want to see it. How do they see it? So, for that, we are also we can leverage the data clean room and the marketplace uh for exploring for
[09:34] collaborating even outside, we can leverage the marketplace and all of these are done by agents. It is a completely agentified solution and
[09:46] along with that we are also taking care of the 100% audit trail wherein every request and response that comes to an agent is logged in lake base. It can
[10:00] either be in a delta lake or it can be in a lake house given the new announcement and launches today we might have to figure out where to store it but we can have 100% audit trail of this.
[10:13] And along with that we got we can also leverage the data bricks AI gateway which will be able to do the governance from a security perspective as well. So this is on a end-to-end
[10:26] agentification. This is this is a data value chain jumpstart that we can give. I will this is the UI that we have built for this again in data bricks apps. So these are some of the agents where you click
[10:38] on it and you can just access the respective agent hub. Moving on just diving a little deeper on a few of the agent hubs. So the discovery and analysis hub. So here
[10:53] we already discussed on all the data modeling data analysis the EDA everything are separate agentic solutions and these agentic solutions are all multi-agentic solutions. So this
[11:05] is automating the data profiling for every format. And we are also like automating or agentifying the data cataloging. And we are also agentifying the
[11:18] entity mapping and the attribute. So this is essentially what the data analysis agents hub does. Moving on. This is another interesting agent hub
[11:31] this is self agent self-healing agent hub. So, we have done this for quite a few customers of us and here this is something that now we can do it using the Genie
[11:44] zero ops that was launched today. So, this is something that we can do using a multiple other options even within Databricks. We have done this using agent breaks. We have done this using Genie and now Genie zero ops would
[11:57] be the ideal solution for this. So, any data quality issue that is something that's not a data engineer should be worrying at all. It is something for the agent to take care of and the data
[12:10] engineer should just monitor and operate the agent. And that is the idea of this self-healing agent uh which we call it the data trust agent hub. And this is not just about data quality. It can be
[12:24] expanded to many more depending on the individual pain points that the client has with Databricks with the data.
[12:37] And this is with the data engineering agents as well. So, for the data engineering agents, we have the LakeFlow designer which is an agentic solution. And here what we can do is we can automate the entire analysis of the data
[12:49] source, map it to the target, the source to the source to target mapping, and also uh take care of the exploratory data analysis initially and then based on that map it to the bronze, silver, and
[13:03] gold layer, do the transformation, do the engineering, and what and the we should not be going and telling the agent like what is the transformation that you need to do. The transformation that needs to be done will be identified
[13:16] by the agent based on the initial document. So, the holy grail is the document that we give initially and that uh holy grail will determine what is going to be the uh transformation that
[13:28] is going to be done, what is the engineering that is done, what is a schema for the bronze layers, silver layer, and gold layer, and what is the schema for the metrics view. It everything is dependent on reverse
[13:42] engineering from the documentation that we get initially. So, that is the holy grail, and we have the multiple agent hubs will decide how every individual
[13:54] agent is going to work and define that particular task objective of that agent. And this is what is going to define how long an
[14:06] data engineering work is going to take, whether it's going to take a day or an hour or a month or a week. So, that is dependent on how well we define the objective of the agents.
[14:21] And the process flow. So, typically, this is the process flow for any of the agent hub. So, this is the data discovery agent. For this, the user creates a template and a sample data
[14:35] file. So, the data discovery agent has a certain template with that we can give, and to that template we give the data. And the template is configurable and data is obviously dependent on the
[14:48] particular domain. So, these two files are given as the input, and the data discovery agent will start questioning the user. So, uh it will analyze the data, it will analyze the template.
[15:01] Based on that, it will start questioning the user on what is the based on the response to it, again to the response, it will also give you options of is it what you're say is this what you're saying? And give you options. It can be
[15:14] a uh choosing one answer or multiple answer. And based on these answers, it will generate the summary and the metadata and that metadata we will be able to
[15:26] approve it or say no to it or we can redo the questioning problem again and based on that the based on the feedback the final summary and metadata is generated and the feedback that metadata
[15:41] generation is also dependent on the knowledge base that it has. So based on the other metadata that it has generated for this respective this respective problem this respective domain within a particular enterprise
[15:54] based on the context it gets input from that and the it also gets input from the data catalog the other agents as well and based on these inputs and the feedback we get the final
[16:08] summary and metadata. So this is the end end to end process flow of a single discovery agent.
[16:20] So essentially this is agentification is a now decision. We have to start looking at agentification now and that doesn't need a new platform with the launches that were done today
[16:32] we have almost everything within data bricks. All we need is a new layer and human in the loop is the architecture that we have. So that is what is the main crux of this what we discussed
[16:46] and we have quite a few agents that are there in the live demo in booth 630 as part of Cognizant. Feel free to join us and write to us in case of anything. Any questions?
[17:01] Thank you. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.