Sponsored by: Deloitte | Generating Realistic, High-Fidelity Synthetic Financial Data on Databricks

Summary

Deloitte's dual-model approach combines a Wasserstein GAN with gradient penalty for structured data (demographics, creditworthiness) and an LLM for behavioral insights like price sensitivity, generating high-fidelity synthetic financial data without exposing sensitive customer information.
This video addresses the compliance bottleneck where accessing real financial data for AI model development requires around three months of regulatory and legal review, which Deloitte's synthetic data accelerator resolves by generating realistic training data that preserves statistical properties without re-identification risk.
The solution is built on the Databricks Data and AI platform using Auto Loader, feature engineering, and GPU training, and produces 500,000 synthetic records in 120 minutes with distributions matching production data, evaluated using RMSE, Earth mover's distance, and nearest-neighbor distance ratio.

Chapters

FAQs

Why is accessing real financial data for AI model training so difficult?

This video explains that financial data is subject to strict privacy regulations requiring data use agreements, regulatory compliance reviews, and provisioning approvals that can take approximately three months before developers can access production data in notebooks. This compliance bottleneck manifests as a technology barrier that slows AI innovation in financial services, particularly for use cases like fraud detection and risk management.

How does Deloitte's dual-model approach generate synthetic financial data?

Deloitte uses two complementary models: a Wasserstein GAN with gradient penalty to generate structured data such as customer demographics and creditworthiness indicators while preserving rare fraud signals in the data distribution, and an LLM to generate synthetic unstructured behavioral data such as advisor notes and meeting records. Together they produce synthetic datasets that replicate both the statistical properties and contextual richness of real customer data.

What evaluation metrics does Deloitte use to validate synthetic data quality?

This video describes three evaluation approaches: RMSE to measure how closely synthetic data distributions match the original, Earth mover's distance (EMD) to compare the shape of distributions between real and synthetic data, and nearest-neighbor distance ratio to assess re-identification risk by measuring how similar synthetic records are to real individuals in the training set.

What results did Deloitte achieve with the synthetic financial data accelerator?

According to this video, the accelerator generates 500,000 synthetic records in approximately 120 minutes, with the resulting distributions matching production data characteristics closely enough for use in fraud detection, risk algorithm testing, and market simulation. The approach reduces the compliance review timeline from months to near-zero for model development while maintaining privacy safeguards.

Full transcript

[00:09] And welcome everybody. So, I'm Hemant. Let me introduce myself. So, I'm an AI and D engineer at Deloitte. So, predominantly working with financial sector and financial services. Um so, today I want to talk about a problem where most of us might have faced across
[00:23] the financial space when we are talking with various AI and ML teams. So, you have a great model idea. You know exactly what data you need in order to build your model and train your model.
[00:37] And you know, you have to wait, you know, um a lot for getting all the legal reviews and compliance reviews to happen in order for even the developers to start developing the code and accessing
[00:50] the data through the notebooks. So, that is our main uh theme here. So, we are trying to So, we tried solving it using this synthetic data generation accelerator. So, what it does is
[01:05] um So, essentially uh just a disclaimer. Uh so, this is currently being um built as an accelerator and we are packaging it up as a uh
[01:18] endpoint or else like an application that we can readily deploy in production. So, it is uh tend to change in the next few months. So, we are still finalizing that uh approach. So, right
[01:32] now uh So, the main business problem is whenever we talk with any financial experts, any financial clients, the main thing they keep saying is, "Yes, we want to build this model. We want to build
[01:45] use this data, but we don't have the data. We cannot get to the data." That is the main problem. So, essentially this is like a governance and you know,
[01:57] uh compliance bottleneck, but it is manifesting as a technology bottleneck, and which is like hindering the AI innovation, especially in the financial services. And financial services is a bit the AI innovation in
[02:11] financial services is a little bit closer to the healthcare sector as well. So, this could be applied across both healthcare and financial services. So, we have built this as an accelerator
[02:24] that can be plugged into any client space. Yeah, like as I mentioned, it is like the restricted data access, where, you know, we need around 3 months of regulatory compliance build data use
[02:38] agreements, and also like provision to get access provision for all the developers to even run a notebook on the production data, or else do like any production back copy, you know, which is
[02:50] going to take forever to get all the approvals. So, that is where we are facing challenges in the especially coming back coming from my background of fraud detection and fraud prevention modeling. So, we face this
[03:03] challenge on a day-to-day basis. So, yeah. And main thing is even if we do any minor change to the columns or any metadata or any data values, it will
[03:15] lead to a like, you know, end number of audit and regulatory trails, and you know, getting you know, it will lead to a lot of compliance damage within the firm. And next is
[03:28] especially for fraud detection and fraud detection and credit decisioning tools, like where we need lot of data
[03:41] in lower environment to rigorously test them end-to-end, which is not possible quite possible all the times because we cannot just pull the whatever data we
[03:53] want from production because it has PII or regulatory and also we cannot just capture you know, all the data into lower environment. We can at max most of the teams that I have seen so far, we are
[04:06] like they're using like sanitized the data like in small batches. They, you know, pull the do a production back copy of let's say few millions of records and they just train the models in lower environment.
[04:20] Um so, but one problem here is the model performs great with this small set of data set and you know, like it keep, you know, like everything looks good and you know, the metrics, the numbers, the
[04:32] precision, recall, whatever the numbers are, uh you know, as long as it meets the enterprise thresholds, they call it a day and they deploy or promote it to production. But the moment we deploy to prod, that's where we start seeing the
[04:46] model to underperform significantly and you know, it will uh uh basically collapse the entire uh use case. So, in order to build trust and
[04:58] also build confidence in the model um performance and the model training, so this project will this accelerator will be a key player and also this will
[05:11] uh all the data availability like the you know, technically speaking, uh your model data model quality depends is a function of your data volume and fidelity. So, it's
[05:27] essentially all the nuance and rare signals have to be present during your training phase so that your model is as as robust as possible. So, that is where
[05:39] this this is like blocking the data access is blocking the AI/ML innovation especially in the financial space. Um yeah. And our solution is uh what at Deloitte
[05:52] what we have built is we have built a dual model approach here. So, what it essentially does is um it's a combination of both
[06:05] GANs, like generative adversarial networks, as well as LLMs. So, what it um helps is it helps in generating on-demand access to realistic synthetic
[06:17] data. And also we can start training and validating your production models or like dev newly development models in lower environment without any PII exposure. And it accelerates the time to
[06:33] fully test your model and also get all your uh compliance overhead uh to be removed during the training phase. So, that's the value add this uh accelerator is going to bring in.
[06:46] So, uh two things. Uh you know, like there are certain scenarios, like certain uh you know, where we can just ask today, you know, cloud code and say, "Hey, generate a data." But the problem is the
[06:59] LLMs can't capture your structural um statistical structure in your data and also the rare signals, especially coming from fraud background, fraud detection and fraud prevention background, where
[07:12] we don't see fraud patterns being captured properly. And we can if we capture as much as fraud patterns, that is going to uh bring in
[07:24] uh reduce the time in the model development and model training, you know, uh once we deploy the models, you know, every week and every literally like I saw people who are like
[07:36] retraining their entire model within a week because there is a new fraud pattern, because there are like lot of users, fraudulent users who try to hit different applications across consumer banking, retail, health care.
[07:51] So, there are like different use cases where like people come up with in very new ways of creating fraud patterns, like bot attacks. So, all those are like being captured as a signals, risk signals. So, that is where this dual
[08:03] model approach is what we have implemented and which is solving for. And so, at a high level, this is like the overall architecture that we have implemented on Databricks. So, essentially we took the internal
[08:17] financial data sets within Deloitte and we started we used auto loader and we ingested the data into the bronze layer and then we
[08:29] prepared the data set did a lot of transformations that are needed and we built features like consumer pricing, consumer confidence index and also then once we build all the
[08:43] features that are necessary and you know, we have like solid set of pre-processing and you know, handling the missing values and all those steps that are done, we started encoding the features, like especially
[08:56] we used GMM based encoding, Gaussian mixture model encoding. So, this is one thing that helps us in capturing the numerical continuous features in a in a better way and then
[09:09] we passed it the prepared data to the encoders. So, we used GAN library, adversarial networks library, auto encoders. We used like standard auto encoder or dual auto encoder depending
[09:21] upon the categorical compressions we want to go with. And then finally, this is where the our dual model approach gets kicked in. So, what it actually does is first we pass it
[09:35] that data, I mean, the GAN library, it's essentially running on Databricks GPU cluster, uh you know, via PyTorch. And um so, this essentially
[09:47] uh captures the it synthesizes the data into a latent space, and based on that, it captures the nuanced fraud and any behavioral
[10:00] patterns in the data, and it essentially helps us in um doing thorough um
[10:12] multimodal continuous features. So, basically, what it means is any kind of compression or embeddings that are happening here are all like standardized, and we are using this GAN
[10:24] library, which would it is which is called as Wasserstein GAN with the gradient penalty. So, basically, this constraints uh loss function, which helps us in
[10:37] um you know, making sure it is capturing all the rare patterns in the data. So, that is the main capability of using this Wasserstein GAN with gradient penalty. And also, once we generate the
[10:52] tabular data, like regular structured fields, for unstructured fields of data, like meeting notes, especially in financial space, like advisor meeting notes or any kind of interaction notes,
[11:04] we are then uh this is like a parallel approach. Uh all the tabular structured fields are being fed into the GAN library, and all the unstructured fields are being fed into the LLM
[11:16] um uh to generate the synthetic notes, synthetic uh interaction notes, or any kind of uh you know, meeting notes with the clients. And also, there is no
[11:28] fine-tuning here. There is no separate vector vector store. It's all happening in memory and in house essentially after that we are evaluating the generated synthetic data using
[11:40] various statistical metrics. So one is this root mean square error. So this is helps us in capturing the categorical distribution whether the distributions are aligned and accurate according to
[11:53] the real data and also we we are also capturing the EMD and the stain. So basically this is like the numerical distributions are being captured or not. So this is like the
[12:05] evaluation metrics. I don't want to go into much details but yeah these are like standard modeling evaluation metrics in GenAI and NNDR this nearest neighbor distribution distance
[12:18] ratio. This helps in capturing the privacy. So what it means is tomorrow once I generate the synthetic data I don't want it to be remap or to be re-identified to a
[12:30] particular customer in production. So that's why like this NNDR feature it should tell us the distance between the actual record versus the synthetically
[12:42] generated record. So we see a significant numbers there as well where we are hitting all the good numbers to make sure the synthetic data that is generated is not being re-identified or
[12:55] tagging getting tagged to any individual. And then finally we are uh loading the data into the gold layer where this is being fed into a risk model or a fraud detection model or a
[13:07] credit decisioning model. And yeah this is like the overall architecture that we have implemented. This is like an internal accelerator that we have built at Deloitte. So we are essentially utilizing it at couple
[13:19] of financial clients to generate synthetic data based on the production data that they already have. Um So, yeah. And one more thing here is uh we should
[13:33] understand is many people they keep saying, "Oh, let's create dummy data in uh lower environment." So, the problem with that is dummy data doesn't capture all the um nuances.
[13:46] And it doesn't capture all the especially fraud patterns. Like there'll be like ton of features, you know, it does or not capture all the distributions and intra correlation intra correlations between all the
[13:59] features. So, that is where this accelerator will uh bring in value to the clients. And um yeah. So, the main um as I said um this uh this is like a dual model approach. Uh one is this
[14:13] Wasserstein GAN with gradient penalty. It preserves the rare signals. Like it captures the fraud anomalies, edge cases, credit profiles, and any other long tail transactions. And we use dual
[14:27] encoder autoencoder. It compresses like sparse categorical features like merchant codes, product types, geography, what not and what not. So, that is number two. And number three is
[14:39] GMM encoding. It captures like multi-model uh continuous features. So, like income, balance, transactions values. This is like a GMM based uh continuous uh feature encoding.
[14:52] And yeah, the GAN preserves the intra feature correlations, income to credit, as I said earlier, uh consumer confidence index, consumer pricing index, pricing sensitivity, which are
[15:05] These all are like core critical features in the financial space which are which needs to be well defined, which needs to be well distributed. And so that, you know, the model that is
[15:17] being trained on these features is capturing the real data that is close to I mean it's capturing the close data that is real closer to the production data.
[15:29] And next um the LLM generates like possible customer interaction notes. So, this is where the unstructured data for unstructured data we are using an LLM. We are doing an
[15:42] LLM call and where we are essentially asking the LLM to uh generate like uh customer interaction notes, advisor memos, meeting notes, and we have the validation metrics as I said earlier.
[15:55] So, currently this is like implemented as a Python plugin and then as an accelerator which can be plugged in into any production environment and we can uh literally generate synthetic data with uh within minutes. So, this is how
[16:09] it is happening. And um the overall business impacts is it will accelerate the AI development across financial space. And it reduces like months of uh no need
[16:23] to go through the months of legal review and PII handling because even if you get access to PII data I mean even if you get access to the production data
[16:36] there will be like lot of regulation across different countries because we are dealing with clients who are like across Europe, UK, you know, where there's lot of data regulation, data policy, and even if you get access to
[16:49] the production data, they will allow only like certain fields to be moved from one environment to another environment and certain constraints have to be followed. So, that is where that is like hindering the overall AI innovation and and it
[17:02] doesn't capture the entire nuances in the data. So, especially for fraud detection which is really needed for fraud detection and mitigation in banking and financial space.
[17:14] So, yeah, that's uh overall theme of the project and yeah, like these are the results. So we have essentially tested this um um we have validated the this
[17:27] accelerator against against uh 350,000 sample customer records um like financial records along with demographics uh cu- cu- customer consumer index,
[17:40] pricing sensitivity, different various demographic features. And we have synthesized 500k records within almost around 120 minutes on Databricks platform. So this is where this is going
[17:54] to be a uh quick plug-and-play. You just plug it in into your production environment and you can just generate synthetic data. So without losing any uh
[18:06] data distributions or any structural integrity in your data or any intra-correlations between categorical versus numerical or any other features. So all those are maintained. So that is
[18:18] going to bring in more value for your model training and also your model performance will be good even once it gets deployed into production. So this is the overall
[18:30] uh uh performance. So here if you see this is like the 50th percentile of LNDR. As I said earlier, synthetic data is equally distant from real customer's data. So it prevents
[18:42] re-identification. So it is the distributions are the same as uh in production data, but the identification of a single record to a production-grade
[18:55] production uh system of record is not possible. So that is the value-add we are bringing in. So it's all synthetic data that's being generated in uh and it can be used for your model uh training
[19:08] in lower environment. Um yeah, and also yeah, we have also written published this as a white paper in healthcare space. So, essentially what we are trying to do there is in
[19:21] healthcare, we are building we are using this for training on healthcare electronic health records. It's basically having uh, patient's vital information. It It's
[19:34] having, uh, you know, doctor notes and it also it's, uh, having like, uh, diagnosis codes. So, we are essentially using this accelerator across various use cases
[19:48] across healthcare, financial clients, and to accelerate the AI development. And this is like the overall, uh, value add that we have bringing and there are like we are trying to package this as, uh,
[20:03] end-to-end, uh, service which can be deployed as well. So, like that's like a work in progress, but yeah, uh, this is like the overall, uh, implementation that we have done at Deloitte.
[20:15] Thank you all. Thank you for joining and I really, uh, appreciate the for you to attend this event. And, uh, yeah, looking forward to connect with you guys. If anyone of you have any,
[20:27] uh, questions or if you have any more inputs or suggestions, definitely I'm looking forward to it. Uh, thanks everyone. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.