Building AI Agents for ETL: Navy Federal's Automated Migration Framework

Summary

Navy Federal Credit Union's AI-driven migration agent saved 2,200+ hours of manual work by automating the generation of configuration files, parameter files, and custom ETL code across 200+ source systems, each of which previously required 2–5 days of manual configuration.
The agent uses prompt engineering and a contextualization strategy with human-in-the-loop validation to automate artifact generation while building data engineer confidence throughout the migration process.
Navy Federal's enterprise lakehouse manages over 5 petabytes of data and thousands of pipelines for 15+ million members, with Unity Catalog deployed in two deliberate phases to establish centralized governance without disrupting active workloads.

Building AI Agents for ETL: Navy Federal's Automated Migration Framework

Migrating hundreds of data pipelines to a new Unity Catalog-enabled framework requires massive engineering effort. Navy Federal faced this challenge: 200 source systems with unique transformation logic, each requiring 2-5 days of manual configuration, totaling over 2,200 hours. The manual process involved reading logs, writing configurations, and debugging, tedious work that slowed modernization.
this video shows how Navy Federal built an AI-driven migration agent using prompt engineering to automate artifact generation. You'll learn the agent architecture, contextualization strategy, and how careful prompt engineering automated configuration files, parameter files, and custom ETL code. Results: 2,200+ hours saved and faster platform modernization. We cover design patterns for AI-assisted data engineering, lessons learned about context importance, human-in-the-loop validation, and building data engineer confidence.
🤝

Chapters

FAQs

How did Navy Federal automate ETL migration to Unity Catalog?

Navy Federal built an AI-driven migration agent that uses prompt engineering to automatically generate configuration files, parameter files, and custom ETL code for each source system. This agent-based approach replaced a manual process that required 2–5 days per source system, saving over 2,200 engineering hours across 200+ source systems.

What is human-in-the-loop validation in AI-driven data engineering?

Human-in-the-loop validation means that while AI agents automate artifact generation, data engineers review and approve the outputs before they are deployed to production. Navy Federal incorporated this pattern into their migration framework to maintain quality standards and build data engineer confidence in the agent-generated artifacts.

How large is Navy Federal's enterprise lakehouse on Databricks?

Navy Federal Credit Union's enterprise lakehouse manages over 5 petabytes of data, thousands of data pipelines, and thousands of tables and files. The organization serves more than 15 million members, making reliable and fast data access foundational to every product decision, risk model, and member experience initiative.

What were the two phases of Navy Federal's Unity Catalog implementation?

In Phase 1 (Q3–Q4 2024), Navy Federal established Unity Catalog as the central access point for all data assets by cataloging data in place without any data movement, delivering immediate centralized governance without disrupting active workloads. In Phase 2 (early 2025), they assessed the effort required to migrate pipeline configurations to a Unity Catalog-enabled engineering framework and began the agent-driven migration.

Full transcript

[00:08] Hey, good afternoon everybody. I want to do a quick introduction. Uh I am Krishna Kumar Suzu Brahmanan, AVP data and analytics engineering. Hi, good afternoon. I am Ansu Bobby, manager data analytics engineering at Navy Federal Credit Union.
[00:23] Yeah. Hey everyone. Uh Sataraasardi. I'm a managing director taking care of cloud AI and data modernization along with other leads who are in this room. Uh I'm also the engagement partner for the project that we are going to review today.
[00:40] Thank you. All right. So here is where we are headed in the next 40 minutes. We will start with a quick introduction go over enterprise uh lakehouse then go through
[00:58] the unity catalog enabled engineering uh framework. Uh from there we will contrast where we started with the manual migration process with the agent-driven
[01:15] approach we built including prompt engineering. We will close with results and impacts we have seen and the best practices and lessons that we learned along the way. So you will be able to take something
[01:31] actionable back to your own teams. Our goal is simple. We want to show you how we went from manual migration process to an automated one and what it took to get there.
[01:51] All right. So before I do a deep dive into the architecture, I want us to ground I want to ground us in the business reality that made this work necessary. Navy Federal Credit Union serves more than 15 million members. Every product
[02:08] decision, every risk model, every member experience insights begins with data. Starting in 2019, we embarked on a multi-year cloud transformation not as a technology exercise but as a
[02:25] strategic investment in how we serve those members. Today our enterprise lakehouse manages over five pabyte of data, thousands of data pipelines and thousands of uh tables and files. So
[02:40] those of you managing data platform at scale know that volume alone doesn't create value. The value comes from how quickly and reliably your uh teams can turn the data into decision.
[02:59] Our journey to unity catalog unfolded in two deliberate phases. each one unlocking the next level of business capability. Now phase one in Q3 and Q4 of 2024, we established Unity
[03:16] catalog as the central access point for all data assets in the lake. There was no data movement. We cataloged in place. The business value here was immediate
[03:34] centralized governance across the entire data estate without disrupting any active workloads. Phase two. In early 2025, we ran an assessment to understand the efforts required to migrate configurations and
[03:50] transformation logic from the legacy data engineering framework to the new Unity catalog enabled framework. So this is where the engineering challenge lives. hundreds of pipelines with unique
[04:07] transformation logic each requiring careful migration. We then redesigned the engineering framework. We rebuilt it for optimized performance, lowering cost and full native integration with Unity catalog.
[04:24] So this is about long-term operational efficiency and positioning the platform for AIdriven workloads. So here's a part that brings us to the heart of today's conversation.
[04:42] During phase two, we started with fully manual migration. Our data engineers were spending two to five business days per source system reading logs, handwriting configurations and
[04:57] debugging. Multiply that across 200 source system, you begin to see the scale of the challenge. So we asked ourself a question. What if we could take the pattern recognition and artifact generation that our best
[05:14] data engineers do intuitively and encode that into an AIdriven migration agent. That is exactly what we did. In the next several slides, we will walk you through the architecture, the prompt engineering
[05:29] and the measurable business impact of the agent. We will show you how we went from a manual migration process to an automated one and set to save more than 2,000 engineering hours in that process.
[05:51] So now you have the context of our transformation journey. Let me show you the data platform that sits at the center of it all. Understanding the platform scale is essential to appreciate why an AIdriven
[06:06] migration is not just desirable but necessary. Starting on the left, we ingest from over 200 source system. These are not trivial flat file feeds. We are talking about relational databases, enterprise
[06:24] applications, semiructured file systems and even social media streams. Each source system has its own unique schema, a unique cadence and unique data quality profile.
[06:39] So that heterogeneity is the first engineering challenge. The data then flows through the engineering layer powered by datab bricks where over a thousand pipelines handle data ingestion, transformation and processing.
[06:59] Each data pipeline enforces three critical controls. Data quality check, data standardization and transformation. These are not optional add-ons. They are embedded in the data engineering uh framework.
[07:16] When you are serving a financial institution of our scale, data integrity is non-negotiable. Moving to the center, this is where data becomes intelligence. Our data science and MLOps layer host over 100 models, a
[07:31] feature store with more than 5,000 attributes and integrated model scoring pipelines. Think about what that means in practice. Member risk profiles, fraud detection models, member insights, all running on
[07:48] well-governed highquality data flowing through the platform. On the right side, data then surfaces through the semantic layer where it becomes business ready, trusted, and secure. The semantic layer serves
[08:06] roughly 3,000 daily users across uh 20 business units. Whether an analyst running an ad hoc SQL queries or an application consuming analytics data or an executive reviewing
[08:22] a dashboard, every interaction traces back through this platform. Underpinning all this is the datab bricks lakehouse built on medallion architecture. Data flows through three progressive zones raw, integrate and
[08:40] curated. Each adding a layer of refinement and governance. That layered approach ensures that uh data quality improves at every stage of the process not just at the point of consumption.
[08:57] Now here is a critical takeaway. The hundreds of data pipelines, configurations, parameter files all have to be migrated to the Unity catalog enabled framework
[09:13] manually. This is an enormous engineering task. That is where we built the agents that we are about to walk you through. I will hand over to Ansud to walk you through how the data
[09:29] engineering framework is structured. As KK mentioned in the introduction, we rearchitected the ETL framework mid2025.
[09:46] Let's take a closer look on all the key activities involved in the transformation. Managing multiple legacy frameworks to source the data from the system of record to the data lake was complex and inefficient.
[10:02] To address this, we have developed a consolidated framework to standardize, simplify and streamline the injection process. This framework not only standardizes the injection process as well as reduces the
[10:20] resource consumption. how across the compute network and storage layers. So by using a single datab bricks cluster by reusing the same spark data frame
[10:35] across the processing stages it minimizes the data eress as well as improves the overall efficiency.
[10:51] N federal's enterprise data lake as the organization's core data platform. It is used for data analytics and data scientist use cases. It follows the Medallian architecture and the data is stored in files and
[11:07] delta table formats. This framework is a configuration and parameter-driven a set of common code libraries used to move data from the source system to the data lake.
[11:22] It is fully compatible with the Unity catalog and leverages UC capabilities like external tables, volumes, target catalog schema.
[11:44] As we transition from the current ETL to the new UC compatible framework, the first step was to generate three types of artifacts. Configuration file, parameter file as well as the custom ETL. Configuration file holds the metadata
[12:01] for UC registration including the source name, path, location, target, catalog and schema tables information and parameter file holds the job input to move the data from landing draw and curated and
[12:19] semantic layer and the custom ETL holds the userdefined functions and non-standard data formats. Generating these framework inputs was manual and time inensive.
[12:36] For example, like of a complex source system for about 70 tables, we are looking to generate over 210 artifacts to move the data from source system to data lake. So there is a clear need of AI agent to
[12:54] accelerate the artifact generation. I have summarized the key efforts and the complexities involved in the conversation conver conversion to the new ETL framework. I
[13:09] will now pass it to Sat to talk through more about the AI agent which which supported to speed up the migration process.
[13:25] Yeah, thanks Ansu. I hope everybody can hear me. Last row. Thank you. Uh Sansu just walked through the Unity catalog framework. Uh she also walked through the process of how a human goes about setting up the configuration.
[13:42] Right? You look at this, it's it's a very tedious process. And u before I get into okay, I'll show the go to the next slide. So this is the agent-driven process that we had created. uh before I go through and and walk through the different steps involved, I want to ask
[13:57] a couple of questions to the audience. Uh so how many how many folks here are data engineers? Show of hands. Okay, you're going to love what you're going to hear. Okay. Uh so how many of you actually use a framework data engineering framework as
[14:14] part of the work that you do uh for injection transformation among the data engineers here? Okay, you're going to learn something that you can go back and actually use at your workplace, something from data bricks summit. Remember this. Uh, and then I think one last question before I
[14:30] get into the agent-driven process. How many of you are using the genie code that you saw earlier today and and to to generate some of the uh actual data engineering code. So good to know. uh so what we are seeing right I think uh what we are
[14:46] seeing in the industry is two design patterns in terms of AI assisted data engineering uh the first pattern is what I'll walk through where we use an agent to create the configuration required for a data engineering framework uh that's the first pattern right because a lot of
[15:02] the companies already have a data engineering framework and creating configuration for that is a tedious process we are going to show how we can use an agent uh to get to those configurations right so that's the first pattern The second pattern is where uh we use these coding agents uh to look at
[15:18] some kind of a mapping document um whether it's a STM and then use that use the coding agent to generate code right so that's the second pattern that we are seeing I'll talk about that when we get into the best practices right those are the two patterns so now getting back to the agent-driven process u so there are
[15:36] actually multiple steps in this process uh right so our goal like Ansu and KK mentioned is ultimately to arrive at these three outputs. Uh right there's a configuration file which has the source target uh information like all of you right data engineers know that you got to take data from one place to another
[15:52] place and then there are multiple hops as we explained in the architecture right there is a landing raw integration curated so the data has to go through all these different hops right so that's what so we have the ETL configuration which specifies for each hop what's the
[16:07] source what's the target uh the JSON parameter file actually provides the schema associated with each of those hops. So what's the source schema? What's the target schema? When the data goes from integrated to curated, not all the columns are passed and there is some kind of transformation. So that's what
[16:23] the JSON parameter file defines. And then in certain cases, there is a custom uh Python code that's returned right in order to do aggregation or some kind of a date transformation. So that's where the custom ETL code comes. So our goal uh with the agent is to generate these
[16:38] three which is very complicated for a human to do, right? So what we did as the first step there are multiple steps in this process. Uh the first step is the uh context right we call that as we we've shown it as input. This is basically providing the context to the agent. Uh so just like a data engineer
[16:54] right a human looks at all this information looks at the uh so so as we have discussed earlier we are migrating from an existing unity existing framework that's not unity catalog based to a framework that's unity catalog based. Right? That's what we are doing.
[17:09] So in order to do this migration the the engineer actually looks at ADF parameters legacy code configuration tables that's what the engineer does. So what we have done nicely is we have taken all that information and packaged into a selfescribable package.json JSON
[17:26] and provided that as context to the agent right so the centerpiece is the agent right so that that's what we give and then what we also do is uh we actually so the target framework as a certain um the code right so we actually take some of not all the code but some
[17:41] code from the target framework and also the sample output right how the output should be generated how this three outputs should look like we pass that information to the agent also we provide all both of that information to the agent so that's The first step right providing the context so that the agent
[17:57] can understand and act appropriately. The second step is reasoning right so this is where the agent actually thinks through hey I've got so much information I need to generate this output how do I generate it and then it notes down okay hey these are the different uh uh this
[18:13] is how I need to create the configuration the the different artifacts right that's how that's the reasoning part that the agent does just like how a human uh data engineer would do so the third is actually generation right so the the artifacts actually gets generated uh so I'll touch upon the
[18:29] context ing in one more slide in a more detailed manner but as such once the reasoning is done uh the artifacts are generated. So this is a very important step. So we call it as validation assurance where we actually uh it's a self-reflection right it's a it's a agentic design pattern self-reflection.
[18:46] So we do self-reflection of the output. So the agent actually looks at the input again looks at the output and then we do some prompting in prompt engineering so that the agent performs the validation check as to whether the outputs are generated correctly. So that's that's the fourth step and the fourth step is
[19:02] where the human comes in. Uh for example when we did it uh the initial week when we were validating and trying this out uh so we got certain outputs which were not consistent which were not deployable. So it generated some SQL and that SQL we couldn't actually run the
[19:18] SQL and get it into the datab bricks ecosystem. So those kinds of inputs were provided by the human. Hey go back and look at your SQL again. Right? So that that feeds back to the agent context and and from the next time it started to generate it properly. So so this is the
[19:34] learning loop that we provide. So once this this whole process is done uh you get these files and then we actually load it to our dev environment run the ETL to validate whether the configuration is correct and once the uh
[19:49] not the configuration whether the data is uh is correct right from from one layer through the different layers whether the data is loaded correctly and once we have validated that then we promote the configuration to higher environments that's the process that we follow.
[20:05] Okay. So I will I think I'll open we'll open it up for questions towards the end but I'll just go through a few more slides. So so in agent AI right uh context is the program and and how you make the
[20:22] context so that your output is deterministic that's that's the work that we need to do in agent and that's what you have stated in the bottom here. Uh so uh there are three key steps in in in the in setting the context. So the first step is to provide agent an
[20:38] identity. Uh so the identity is basically that the agent is a data engineer smart data engineer. Uh right who who's maybe experienced in writing data bricks code in this case. Uh and and then we also give it a set of rules.
[20:54] uh for example you know because it has to understand the vocabulary around the uh legacy code the new code so we give it a set of rules right so that's the initial identity establishing the identity once that is done um like we saw in the earlier slide uh so we
[21:09] actually pass in the right context in terms of uh for each source so this is done at a source level so for each source we pass the source target mapping uh we pass how many hops are there so all that information that is there in the package JSON we provide that uh to
[21:25] the agent right so that's the second piece of information we give and the third we refer to this as few short prompting in agentic so this is where we specify what the output should be uh so we give it example say uh for uh when you're generating config SQL this is how
[21:41] the output should be right so we actually specify exactly what it should be uh and and and that's what you are seeing here and and actually what happened is this is when you do context engineering It's a trial and error process, right? It's not like you'll get it right the first time. So you go
[21:57] through a few iterations and then you reach a stage where you are comfortable and you always have the human in the loop step where the human can go and correct as well. So so that that's that's how we did the context engineering. So this is the results and impact and
[22:13] this is true. Okay. Absolutely true. So uh so so we are going to save 2,200 hours, right? That's the total effort that we'll save as a result of this project. Uh we are talking about 200 sources as you saw. Uh and and if you
[22:28] look at each of the steps right so whether it's the initial log analysis that the human does uh in order to arrive at the right configuration that's what so human does those different things right the human goes through the logs and then goes to the uh creates the
[22:44] ETL configuration and then creates the parameter files. If a human like I think when we initially designed this process we had the AI engineer actually shadow the human right and and the AI engineer actually felt bad for the data engineer right because the process of creating
[23:00] the CTL configuration the parameter files was just tedious right so so that's how we were able to if you see right for each of those steps we were able to save close to 40 50% of the time the custom ETL is a bit more complex uh right so so that's where I think we have
[23:16] seen that uh there is some level of human review required uh and and I think what what we're showing here is overall we have been able to save uh this much effort and we are able to use that to do other high quality work okay
[23:31] and and these are the best practices uh based on the work uh that we have done uh so the first one is around the unity catalog framework itself so we upgraded from the uh legacy framework to the new framework and new framework uh is much more um uh performant and and cost
[23:48] effective. Uh it uses a single uh cluster uh and then and then it runs through the entire hop all the hops in one go rather than one by one. So that that's the that's one thing that we did. Uh the second is I think when we are doing prompts for example right when we
[24:03] are setting up the identity of an agent uh you you have to externalize it right and this is what I think in the industry we are calling it a skills so that anybody can use the prompts as as we anybody meaning other agent developers can use the same prompt we don't have to
[24:19] again create a data engineer identity right the same prompt can be reused uh the third best practice is around uh templatizing uh your prompts and and making sure that you pass in the right parameters uh so that uh uh the prompts itself can be
[24:36] reused. Uh so this is something that we are seeing in the industry and I think like I mentioned originally when we did this we didn't specify the LLM outputs to be a deployable artifact. Once we said hey make the outputs as deployable artifacts the outputs were much better. So that's that's the best practice we
[24:52] have called out there. And then I think like in any uh agentic system, we got to have the human in the loop, right? Those are the four uh best practices um that I have noted here. In terms of lessons learned, right? Uh I think this is something that we heard from Ali
[25:08] this morning in the keynote. Uh so where he was talking about AGI and how frontier models have become really really smart. So so we got to believe the model. So if you're seeing hallucination it's it's not it's because it's I think like you have said it's a metadata gap. It's not a model flaw. Uh
[25:25] context is more valuable than the model. So we actually tested this on GBT 5.2. Uh uh so so I think that's a very important thing as agent developers that we need to understand right the more context you give it it really really works right I think some of the outputs
[25:41] the artifacts are really very complex and and agent is able to generate that. The second is I think whenever we are doing any data engineering project uh we got to have this agentic first mindset right uh so that's why I think this
[25:56] whole uh specdriven development is is becoming uh the norm as we do more more and more data engineering projects. So that agentic first approach I emphasize and and I think that that's what we have mentioned here. Uh the third is I I I think this is something that we have
[26:12] seen with the work that we did in Navy Federal Credit Union where uh we wanted to make sure that we engage the data engineers the uh the broader data engineering community to get inputs right because if this is done in isolation and if the and and when this
[26:29] work actually needs to get done and if the data engineers are not aware then there is some resistance right so so that's why engaging data engineers early in the process to build confidence in the agent and I think I think that's that's very important right so that's the lessons learned so I'll that's
[26:45] that's that's what we wanted to talk through at least I wanted to talk through I'll invite uh KK to close and then we'll take some questions after KK is closing yeah for the details yeah uh to summarize u today we have uh enterprise lakehouse uh
[27:03] which has uh several pabytes of data thousands of pipelines thousands of tables all now governed through Unity catalog. The hardest part was migrating the pipelines of the legacy framework
[27:19] that where the agent uh agentic approach made the difference. So what does it mean for us as an organization? It means uh a well-governed AI ready platform means faster decision uh
[27:34] sharper member intelligence and uh durable competitive advantage for Navy Federal. So that uh concludes this presentation. I hope the information that we shared uh will be helpful for you in your journey as well.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.