Sponsored by: Neurealm | AI driven Approach to Accelerate Data Modernization by up to 25%

Summary

Neurealm's NeuFlow framework unifies three AI-powered components—DataBridge for intelligent migration, TrustFlow for automated data quality, and MatchFlow for source-to-target reconciliation—into a single no-code data engineering accelerator that makes data platforms AI-ready on Databricks.
This video explains why enterprise data modernization projects typically require 12–24 months of manual effort for code conversion, schema mapping, validation, and data quality remediation, and how an AI-driven approach can reduce migration timelines by up to 25% and automate nearly 50% of quality operations.
NeuFlow targets legacy ETL complexity, schema drift, and manual low-level design that create data quality debt, and its post-migration architecture supports Apache Iceberg for future-ready, AI-ready data with a claimed 20–30% reduction in operational costs.

Chapters

FAQs

Why do enterprise data modernization projects typically take 12–24 months?

This video explains that the long timelines stem from manual activities including code conversion of legacy ETL pipelines, schema mapping between source and target systems, data validation, and resolving data quality issues accumulated over years of production use. Additionally, disconnected tools for migration, quality, and reconciliation create handoff complexity that adds further delays and coordination overhead.

What are the three components of Neurealm's NeuFlow framework?

NeuFlow comprises DataBridge, which handles intelligent automated migration of legacy workloads including ETL code conversion and schema mapping; TrustFlow, which applies automated data quality rules and uses agent-based remediation to address quality issues; and MatchFlow, which performs source-to-target reconciliation and validates business rules to confirm that migrated data matches expected outputs.

What results does Neurealm claim for organizations using NeuFlow?

According to this video, Neurealm claims that NeuFlow can reduce migration timelines by up to 25%, automate nearly 50% of data quality operations that previously required manual effort, and deliver 20–30% lower operational costs after migration. These figures are presented as targets for organizations modernizing from legacy ETL and data warehouse platforms onto Databricks.

How does NeuFlow ensure data is AI-ready after migration to Databricks?

NeuFlow includes a post-migration readiness layer that supports open table formats including Apache Iceberg, ensuring the migrated data sits in a format compatible with Databricks AI workloads. TrustFlow continues to monitor and enforce data quality rules after migration, and the unified migration dashboard provides visibility into quality gates and readiness status so teams can confirm data is fit for AI use cases before deploying models.

Full transcript

[00:08] Hello everyone. And thank you for joining the session today. I'm Pragatheesh, part of data and analytics practice Newrlum. Today I will be talking about how organization can modernize the data
[00:20] platform through AI driven engineering approaches and data bricks. Across industries, enterprises today heavily investing on cloud analytics and AI initiatives.
[00:34] But one challenge is remaining common across everywhere, that is modernizing the data platform is still slow, expensive, and manual. Many
[00:47] organizations spend months, sometimes years in terms of moving the legacy ETL pipelines as well as the data into modern platforms. And preparing the trusted data on the
[01:01] AI initiatives. At the same time, the expectations from the management or board is little higher. They wanted to migrate the data much faster, they wanted to get the insights faster, and
[01:14] also they wanted to make the platform AI ready as soon as possible. So, the question that we started asking ourselves is what if AI could assist in terms of
[01:27] modernizing the data platform itself? What if AI could assist in terms of the reduce the manual engineering effort across the modernization journey in
[01:39] terms of validation, in terms of schema mapping, conversion. That is what I'm talking about today.
[01:51] So, before we getting into the problem statement or solution statement, we would like to This I struggle today despite large modernization initiatives, one thing that we are seeing in common is the AI initiatives, whether it is
[02:05] success or fail, it based on the underlying data set. The enterprises currently always takes the modernization product at least 12 to 24 months because of lot of manual activities such as code
[02:19] conversion, mapping, validation, data quality issues. AI initiatives fail due to poor trusted data, not because of the AI models that runs on top of the data. And moreover,
[02:33] the discon- disconnected tools which they are running currently because the the data warehousing platform currently is having your ETL component, the data warehouse data store, and then the BA
[02:45] component. So, the disconnected tools create operational complexity. That's where we started to think about what if the modernization itself become AI driven rather than the
[02:58] the consumable data sets, how the AI driven approach helps data engineers to modernize the platform much faster.
[03:13] So, before getting into that, what we have actually inferred from the industry pain points is the legacy ETL complexity. What it means? Over the period of time, several data
[03:27] stored procedures, for example, consider me, I have actually written a stored procedures 20 years back. Now, there is a no documentation or over a period of time, the change through change request, the procedure has been
[03:40] evolved a lot. Now, if they wanted to move this into a modern platform, it has a huge dependency of the underlying documentation and the procedures written by the engineers, right? That causes the
[03:54] slow migration. The next one is manual data quality checks. When you actually get into the platform, you start understanding what exactly the data quality issues in terms of the source and target mapping, as
[04:08] well as the uniqueness, null checks, constraints. All these are all manually done by the data engineers. That actually goes for high operational costs. Under the siloed governance is nothing
[04:21] but you are maintaining multiple ecosystems as part of your data warehousing system, which actually creates lack of trust because you are moving the data from one place to another place, which actually needs governance.
[04:37] Then the next one is a schema drift, where the currently the system, due to the evolution of the data era, the challenges is the source systems is keep on upgrading, so the data pipelines that you are building, which needs structural
[04:50] changes on the target system, which needs to be corrected. Because of that, the pipelines has been broken because the new platform has the schema evolution feature, whereas the traditional platform does not have that.
[05:03] And finally, you have another data Lack of lineage is nothing but since you have a siloed governance, automatically it goes towards lack of lineage, which needs AI compliance risk. Because the AI
[05:17] compliances request the traceability towards the backward. So, which actually lacks as part of the platform. So, what we are trying to say here is the traditional modernization is too reactive and too manual.
[05:37] So, why the existing approaches delay the traditional modernization cycle, which has seven phases, why actually it gets delayed? The first phase is assessment, which actually goes toward the source system
[05:49] analysis, which actually understand the inventory of the objects which is sits in the source system. The second and third component, which is mapping and code conversion, is manual activity which we are currently doing in terms of
[06:02] doing your low-level documentation where you are actually doing the translation between your source and target. And eventually, based on the source and target, you are doing the code conversion, whether it may be PySpark or Spark SQL or any other legacy languages.
[06:16] And then the fourth and fifth one is more of a validation, which actually takes care of the source and target data that you moved, which needs to be validated. And the testing is nothing but that the code that you converted
[06:28] from your legacy platform to modern platform, which needs to be evaluated, which needs to be compared. And finally, sixth and seventh is more of afterthoughts. That is nothing but once the migration has been settled, then only the governance and optimization
[06:41] comes into picture. So, the all these seven tracks, which needs to be sequentially done, where majority of the things has been done manual. For example, the assessment is manual, the mapping which
[06:55] we are doing in terms of low-level design is manual process, the code conversion has been done manual process, and validation testing is done by the QA engineers. So, majority of task is involved a human
[07:08] dependency. And the next one is all these human dependency task is again, it is done by repeatedly as part of the process. And the tool fragmentation is nothing but as
[07:21] I mentioned, data quality I mean the for data ingestion you have one tool, for data processing you have another tool, for orchestration you have another tool. So, you have you have to have managed all these tool fragmentation across the
[07:33] product I mean across the data modernization cycle. And finally, the slow feedback cycle is nothing but if you are finding any issues in the testing cycle, then it has to go towards mapping the correction and then the code
[07:47] conversion will happen and then eventually the slow back This is This is nothing but the person who is doing the engineer who is doing the testing is not directly not doing the code conversion or mapping. That is where the slow
[08:00] feedback cycle comes into picture. So, what we have thought is like all these manual activities whether instead of attacking only one area, let's try to attack all these manual activities
[08:13] across the journey. That's where we thought about creating an a framework called New Flow, which is a proprietary to Neural LAN, which is nothing but an AA modernization accelerator, which does all these activities
[08:26] seamlessly in terms of Instead of manual, it is AA AA based as well as automated automation based. So, it has three components. The first one is data bridge, which actually
[08:38] seamlessly migrates the data from your source to target, which involves the schema analysis, the code conversion, as well as the metadata conversion. So, the metadata conversion is nothing but
[08:51] the your your schema mapping. The code was conversion is nothing but whatever the stored procedures or legacy procedures which is you have written. The next component is automatically you have to do a reconciliation, right? Once the data has been migrated, then you
[09:04] have to do a reconciliation as part of it. In a traditional modernization cycle, the QA engineer does all these validation and reconciliation. As part of this New Flow framework, we ourselves
[09:18] has a build a accelerator which actually does this reconciliation automatically for you once the migration has been completed. The The third component is Trust Flow, where the data quality issue comes. Without data quality resolution or
[09:31] without actually the data is perfect to AA ready data will not be there. I mean, the the model that you runs on top of the data will not be seamless and which is more produce the
[09:43] invalid results. That's where the trust flow comes into picture. While you are moving the data, if you find any data quality issues, you can configure as a rules as part of the process and you will settle the data
[09:55] with with more qualified data into for for AI for AI analysis. So, this is actually runs on data bricks which actually uses all the components
[10:08] which I actually listed in terms of it is using open lake Delta Lake and Lake Bridge is the data bridge has been written on top of Lake Bridge which actually in turn calls the Transpy LLM which actually in turn calls
[10:21] Mosaic AI as well as Genie code. And Unity catalog is used to for governance when there is an when you are doing a migration, if you are having a performance issue, we are actually using Photon enable cluster for
[10:33] performance issues. And also, it is actually capable enough to do serverless process as well as part of the migration as well as data quality checks.
[10:45] So, this is basically a functional architecture of new flow which has three-layered architecture. The first one is nothing but the data store where you are having the set of data stores which we are supporting. Whether it may be a MySQL or MS SQL Server, Oracle, or
[10:59] Teradata, or Postgres which you are supporting as a storage layer. The processing and AI engineers where our component sits. Whether it may be data bridge or trust flow or match flow. In
[11:11] in terms of this particular layer, we have embedded all our AI assessment or AI technologies like AutoGen, LLM, uh rag I mean, knowledge base. Everything has been layered in the process and AI
[11:23] engineer layer itself. The third layer is where the actual business comes into picture and they have verify all the results in terms of dashboard or in terms of reports or even chatbot they can query and they can
[11:36] monitor the results. So, the basically what we have trying to say here is the AI has been embedded as part of the workflow not after that.
[11:49] Let's dig deeper on the data bridge side of it. So, as I mentioned the traditional migration has manual mappings, manual code rewrite. They have to understand the existing code and then they have to rewrite the code as part of the new platform and which actually
[12:01] creates a slow validation as well because you have to manually compare the data sets. What we have achieved is we have embedded AI driven architecture within this which automatically understand your metadata. It will recognize the pattern
[12:15] and it will write it will actually generate the transformation logic as part of the LLM process and it actually helps the new person to an onboard and he can easily adopt the particular technology. So,
[12:28] what we are doing is part of the process, we are actually creating the functional document as well. So, when I when I highlighted that the procedure which is written 20 years back has no documentation, right? As part of this
[12:40] process when we are doing a migration, we are creating a functional document of a stored procedure or our legacy platform scripts that before converting into a process.
[12:56] So, this is See, one of the major challenge that organization or business faces is the visibility of the migration where they wanted to track where we are right now in terms of migration process. What actually happens in the industry is the data engineer consolidate all the
[13:09] reports, audits, and validation reports and he can create an Excel sheet or PowerPoint where he can share this data to the management where we are actually tracking the results. So, what we have
[13:21] done is we have created a single unified dashboard which actually gives all these metrics in a single dashboard with six cards. The alerts and notifications is always goes to an email as well as it actually
[13:34] highlights in the platform itself, where we are right now, what kind of issues going on, and it will also gives a migration process in real time. How many How many I mean objects has been migrated, how many objects is in
[13:47] progress, how many objects has been failed, and also along with the data migration quality, which actually gives why this is failed. So, it It also gives the complexity analysis as part of the dashboard. So,
[13:59] what we are trying to say here is when you are giving a dashboard kind of a stuff, it reduce the burden of the data engineers who consolidate and gives this data to management. So, the the process that we are highlighting here is
[14:12] reducing the manual burden of the data engineer, rather than he build a report, he can go and concentrate on the architecture and optimization of the problem statement.
[14:26] The other one which we are actually showcasing here is one is the unified dashboard, the other one is the stored procedure whatever I have discussed, right? Which actually creates a functional document. It is a more of agentic architecture which seven
[14:39] agents talk each other. The first agent will create a functional document of a stored procedure, and the second agent try to simulate in a PySpark code, and the third agent will come and test automatically, then it
[14:52] will migrate the data. As part of that, it will create a test cases also. So, while doing the migration of a stored procedure, you can just you have a screen where you can compare the whether the code has been properly
[15:05] converted or not. Even you can go and you can edit it, and then you can just send this into agentic architecture to verify that process, whether it is equivalent to what we have written as part of the stored procedure, and then
[15:18] you can deploy it. So, what we are trying to say here is you can reduce the manual effort in terms of understanding the stored procedures what which is written in traditionally 20 years back.
[15:34] The next part is more of data quality. So, always we we believe that AI without trusted data creates deceptive results. So, the inconsistent and incomplete data always yield unexpected results as part of the AI model. So, the features that
[15:48] we are actually building as part of Trust Flow is when you give the table, it actually automatically profiles your data set and it will recommend what kind of data quality rules that you can apply on top
[16:00] of it. And if you wanted to apply the rule, automatically it will apply the rule and it will create a DQ score on top of it. Or else, you can it will give a user interface, you can create your
[16:13] own business rules and you can deploy it as part of the process. So, what it actually does is it will reduce the human I mean it it it reduce the data engineering effort up to 55 50% since we have
[16:27] automated everything in terms of business rules. You don't need to create business rules using any language. It is more of a configured screen screen where he can go and he can log and he can do all the necessary business rules
[16:41] configuration. The next one is match flow which actually does the reconciliation process
[16:53] after the migration. So, it has One of the major challenge what we are seeing in the industry is the migration estimation is not at all included in the
[17:05] project estimation part of it. What we have seen in the industry is most of the modernization projects ignores reconciliation pain because they have actually underestimated the reconciliation where they have to compare the source and target as well as
[17:18] they have to incorporate all the business rules as part of the process. So, as part of I mean match flow, we actually do the source to I mean target validation, intelligent mapping, and you
[17:31] can do a threshold based reconciliation as well. When you are having billions of records, you cannot compare billions of records, right? So, instead of that, we will give a percentile where you can say like 5% of records I have to compare.
[17:43] So, what kind of comparison you can do? You can do a reconciliation between source and target. You can do a cell level comparison, source to target organization. All these process you can do and you can achieve through match
[17:55] flow. In match flow, you have executive reconciliation dashboard which actually gives all the metrics.
[18:07] And also we have a reconciliation rule configuration where you can get into that and you can configure whatever the process that I have mentioned, whether it may be percentile based or record count based.
[18:20] So, it uses data data bricks based all the components as I mentioned for Unity Catalog, I mean unified governments, it is Unity Catalog, Delta Lake for open storage. We also have a flexibility to migrate into Apache Iceberg as well as part of this during the I mean as part
[18:34] of the configuration if you do it, it will go for Apache Iceberg as well. And for data quality, we are using declarative pipelines. As I said like since it calls data bridge as part of it, it goes via Mosaic and it actually uses
[18:48] Genie code as well. And we also used open AI for some of the stored procedure conversion. So, these are all the generic use cases which we can resolve using this
[19:00] overall accelerator which we have built. So, even though you have different domains, the problem statement remains same. You have actually have manual process for all these migration, reconciliation, as well as
[19:13] data quality issues. The key benefits is not coming from, as I mentioned, it is 25% is not estimated or not measured. It is observed. The meaning that when you actually reduce
[19:27] the manual effort across all this journey, whether it may be source to target mapping or code conversion or reconciliation or validation, data quality checks. So, overall, if you
[19:39] consider all this process manual effort has been automated as well as agentified, then automatically you are getting 25% of faster modernization. That is what we
[19:51] have highlighted. You can accelerate your data modernization using this. And as part of this, we are seeing 50% of reduction in manual operations. Since 25% you are accelerate faster,
[20:03] what actually happens is when you are running the system parallel, you are reading reducing that 25%, which actually indirectly you are decommissioning the existing system, which actually lowers your 30% cost.
[20:17] So, you modernized. Now, what is beyond that? So, you want to I mean, you once modernized, you have to start monetize it, right? That is where the next feature set comes into picture. So, we are using Delta Share.
[20:30] On top of it, we are actually doing a controlled automation framework, which actually does an approval workflow to share the data beyond your data sets for research and pharmaceutical industries in terms of health care. So, which is
[20:42] Share Flow. The Twin Flow is most of the customers right now asking. So, you modernize the platform. And after modernized, what you are doing is the Spark code which you have written,
[20:54] due to the changes that is frequently coming in the source system as well as requirement, what you are trying to say is you wanted to modify that Spark code in the production system. So, simulate it rather than you do it in
[21:07] dev and QA environment, try to simulate it similarly into the production environment and do this. That is called twin flow. We have actually did a prototype here and we are working on it. Most of the customer asking this as
[21:19] well. And then finally, the intel flow is more of a AI driven a risk assessment in terms of classification. So, it's it's not only any specific domain, across all the domain we are trying to achieve here.
[21:31] So, that's what our future vision. So, if you have any feedback, please feel free to give that feedback and I will be available in booth 666, where you can
[21:44] have anything in terms of the acceleration work and you can feel free and come. Probably, I can show you the live demo. Thank you very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.