Democratizing Data in a Regulated Industry: Best Practices and Outcomes With J.P. Morgan Payments
Summary
- Regulated industry democratization: JP Morgan Payments won Databricks' 2024 Disruptor award for building an internal technology marketplace on Databricks and AWS while maintaining strict banking compliance.
- Data transformation journey: The approach focuses on breaking silos, building trust and creating value through collaboration and federation, with three key drivers: innovation, simplification and democratization at scale.
- Customer-facing value: JP Morgan Payments now develops customer products like Cash Flow Intelligence by enabling end-users to own and innovate with their data while maintaining governance and security.
Democratizing Data in a Regulated Industry: Best Practices and Outcomes With J.P. Morgan Payments

Talk by Narayan Raj, Managing Director, JPMorgan Chase
Join our 2024 Databricks Disruptor award winners for a session on how they leveraged the Databricks and AWS platforms to build an internal technology marketplace in the highly regulated banking industry empowering end-users to innovate and own their data sets while maintaining strict compliance. In this talk, leaders from the J.P. Morgan Payments Data team share how they’ve done it — from keeping customer needs at the center of all decision-making to promoting a culture of experimentation. They’ll also expand upon how J.P. Morgan Payments products team now leverages the data platform they’ve built to create customer products including Cash Flow Intelligence.
Here’s more to explore:
Economist Impact: Unlocking enterprise AI: https://www.databricks.com/resources/webinar/unlocking-enterprise-ai
The State of Data + AI: https://www.databricks.com/resources/ebook/state-of-data-ai
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
What is the JP Morgan Payments platform?
JP Morgan Payments is the payments business of JP Morgan Chase, one of the world's largest financial institutions processing millions of payment transactions daily.
What was the data team's goal?
JP Morgan Payments aimed to democratize data by empowering end-users to innovate and own their datasets while maintaining strict compliance in this highly regulated industry.
What customer products resulted?
This video mentions Cash Flow Intelligence as a customer product created by leveraging the modern data platform built on Databricks and AWS.
How did they maintain compliance?
The platform was specifically designed for the highly regulated banking industry, with built-in governance and security to support strict compliance requirements.
Full transcript
Hi everyone, I'm Naran Raj, chief data officer, payments business JP Morgan Chase. I know this is a post lunch session, so we're going to try and keep it as interactive as possible. Last year we were here at data bricks AI summit where we presented our modern data platform and there were quite a few follow-up questions to our presentation and one of the key question was how did we achieve modernization in such a highly regulated industry also last year we won the disruptor award the data bricks disruptor award and uh I took my whole data engineering 19 to the restaurant to celebrate.
Do you know what did they do? They joined all the tables. so today we're going to talk about democratizing intelligence in the in a regulated industry. We're going to talk about what is it, how did we at JP Morgan achieve it. So first we're and we're going to take you through our journey our journey of transformation where we focus on breaking silos uh building uh trust and um also creating value at every step through collaboration and federation.
We're going to pro uh we we're going to focus on how we provide first cl first first class client service to our clients and uh we're going to talk about our journey I the three key drivers of our transformation journey focused on innovation simplification democratization at scale with a relentless focus on governance security and quality Throughout the presentation, we're going to talk about some of the foundational building blocks.
First, I'm going to take you through what is the scale of our uh of our business. Then go through some foundational building blocks of democratizing data. What does it mean? Then we're going to get into a little bit about the problem statement that we are looking to solve solution and finally we'll get into the architecture. So again we're going to keep it interactive. I wanted to start off with taking a quick pulse across the room.
How many of you here have worked with payments? Huh? That's that's good representation. Let me be specific. How many of you have worked with payments today? All right. I guess some of you know the answer. The answer is I see you. You definitely know the answer. So, uh the answer is most likely everyone in this room has worked with payments today. If you do a transaction that moves money, at the end of the day it is a payment transaction.
Imagine the amount of data that generate that gets generated through every payment transaction across the world. At JP Morgan, we process 10 trillion dollars in payments every day. Generates a lot of data. It is the place to be for data scientists, data engineers, and data geeks like me. JP Morgan payments business actually um so the payments business at JP Morgan provides a range of different services to our clients ranging from payment payment processing to treasury management services.
Some of the key business areas we have is the merchant services for businesses to accept credit and debit card payments. Our treasury services for cash management, payment processing and liquidity solutions with crossber payments across 120 or 160 plus countries, 120 plus currencies. our trade and trade finance and working capital business, our digital enablement business, our embedded finance financial solutions, our API developer portal.
Our mission our our mission at data within JP Morgan payments built JP Morgan payments data intelligence to drive smarter, faster, safer payment services for all our clients. technology, AI, data intelligence underpin the foundation of payments business at JP Morgan. You can learn more at www. jpmorggan. com/payments. Now you have a idea of the scale we are dealing with $10 trillion a day 120 plus currencies 160 plus countries.
Let's talk about some of the foundational building blocks of data democratization. These are non-negotiables. When we started our transformation journey a few years ago, we made sure that we set some guardrails for our organization as we built our data platforms and modernize and transform the data uh landscape at JP Morgan Payments. The first one being security and safety. Security is the topmost priority to safeguard our clients information.
Security in terms of protecting the data. Security in terms of access to data. Security in terms of um just making sure that we have the right uh access level access controls in place. when we started our journey and I some of the some of the protection around encryption at rest and encryption at motion when we started our journey encryption solutions were not that advanced where they were not that performant but over a period of time we've seen technology has advanced and they've become performant and with the evolution of AI we have more support in terms of tokenization anom anonymization and mascul The next is governance.
Governance is more focused on making sure every bite of data that we manage is managed with purpose. It focuses on making sure that we have proper responsible use of data in place. We had to make sure we understood every bite of data that was being processed through our platform and we came up with a governance framework that managed a purpose of that data or the right responsible use of the data quality. Quality in terms of completeness, accuracy and timeliness uh of data.
How many of you in the audience have heard the term gigo or gigo? Yeah, it's become more important with as we seed as we feed more data into AI garbage in garbage out. We have to ensure quality of data. We've built automated rails to actually make sure that we manage quality through controls across the entire end to-end flow of our data. Now the top part is about managing data. The bottom section is going to be about our consumers.
Democratizing data without understanding the consumer persona is incomplete. We needed to make sure our data is well explained, well defined, well described. We needed to make sure that it is accessible for all our consumers to understand the data that is being processed through our system. And finally, focus on value generation. As I mentioned, every piece of information that was being uh managed within the system needed to have a purpose.
And as we built our transformation journey or as we went through our transformation journey, we actually built our platform based on value uh that value of use cases and products that we were building and delivering and most importantly trust. There's a saying trust is built in drops and lost in buckets. In the data world, trust is built in bits and lost in pabytes. So it is very important that we make sure that the platform that we built has strong guardrails to really promote trust in the platform.
As we looked at the non-negotiables, now we start to talk about data democratization at a very high level. What is it? It's two pieces. data, access to data to the consumers. And we talked about the non-negotiables from a foundational perspective about how we need to kind of focus on providing the right data, high quality, secure access to data and making sure that the data is well described. On the right hand side, we needed to understand our consumers from a standpoint of making sure that we have harmonized our data in the platform in a standard language that the d the the consumers can understand through a model.
We also needed to make sure that we have the right entitlements uh management scheme and framework in place so we can manage things like arbback, aback and so on which is roll back roll based access controls and uh attribute based access controls and we also needed to make sure that our consumers had an area where they could use the data in a secure way to innovate and play with it. So, we created our lakehouse playground.
We're going to talk about lakehouse a little bit more. So, again, a quick pulse check. How many of you in the audience know what a lakehouse is? All right, you're at the right conference. underpinning the whole framework or the whole data democratization uh model with very strong technical data quality control in place that focused on completeness, accuracy and timely delivery of data. Now that we've understood what the scope that we are dealing with, we've understood some of the foundational building blocks of democratization.
Let's talk a little bit more about the problem statement. Pulse chat. How many of you in the audience have heard the FOMO effect of data? Wow. Nobody. FOMO. Fear of missing out. All right. Let's try again. How many of you have heard data is the new oil? All right, great. How many of you have heard data is the king? Some of you. That used to be a term I I would say a decade ago. So in our legacy platforms you can see data is locked up in silos on the left hand side in the source systems and every time we needed data we built bespoke fe bespoke pipes to source that data into a a data warehouse and then over a period of time many of these data warehouses cropped up data locked up ping i. e. data locked up rendering data powerless are data being clogged in the pipes i. e. oil clogged in pipes and not being able to refine it to reach it reach its potential value and finally the FOMO effect of data.
Most of these data warehouses were closed systems where change was very expensive and every time you needed to kind of build uh insights it it it was even more expensive. There was no sharing. So we had many metric stores across our data landscape. We had many data stores and of course we had too many pipes feeding these data stores. Now that you've understood the problem, what is the solution? It's the best solution that we've heard.
Build a data lake. We love that, right? It I think it is probably not the right answer or it is not the right answer. It is part of the answer, right? um we had to think about our journey very differently. Of course, we wanted to build a data lake, but more importantly, we wanted to think about our consumers. So, we actually anchored our journey on our consumers. The days of passive consumption are long gone. Data consumers are an active participant in the data ecosystem.
They are shaping the data product. They're shaping new services. They're building new offerings that are coming out of the data data ecosystem. They they now the whole power shift in the consumption space is they have they feel a sense of ownership and belonging within the overall ecosystem. So we had to build a platform that was consumer focused. Our approach was to build a single open platform that supports a variety of capabilities for multi-persona user base.
So the first thing we did was we actually split all our capabilities or we split up all our platform focused on consumer specific capabilities. If you look at the picture from the left to the right the box in the middle is a multiplayer zone or a multi-persona multiplayer platform. We built an ingest framework for data engineers to build uh their data flows. We built a trans transformed framework for data engineers to be able to transform their data.
We built an orchestration framework for engineers to easily federate and build their data flow orchestration. On the right side, we built an API and SQL based connection uh consumption framework for our developers and our analyst to use. And we built an overall M ML operations framework for our data scientists. We even built out of the box telemetry and observability for our data SR teams. That is easier said than done.
As we were going through this journey, we had to also anchor on some of the key drivers. Our key drivers as we were going through the journey. The first one is innovation. I'm sure everyone has heard and this is a very common term be that gets used in the tech industry. You build it and they'll come. That no longer exists. within the payments business. The payment actually the payments business has evolved in the past decade.
It used to be a very batchbased 24x5 business. Today the payments business is 24 by7 real time by at milliseconds payment processing. So we could no longer just build a data lake that was batch based. We had to innovate. When we started our journey, we actually set out to build a streaming analytics platform that will adapt to the ever evolving needs of the payments business. We also needed to make it a multiplayer multi- uh consumer framework or multiplayer multi-consumer platform so that many different personas could could contribute and build the platform in a federated model.
Second, democratization. And you're going to keep you're going to keep hearing this word democratization because we are at the at the at this session to talk about democratization. What is democratization? I I gave you the building blocks, but if I had to say it in plain English, one line, it is making everyday AI possible for everyone in the organization regardless of their role. What does it take? It takes secure access to data that is high quality, reliable, and trustworthy.
It takes making required tooling available to the consumers so that they can work with the data. And finally we have to make it scalable, fault tolerant, testable and um a and reliable. The third driver simplification. Now as we were going through our journey, we were innovating really fast. We were creating a streaming analytics platform. We were moving to the public cloud. What happens when you start to move to a public cloud-based solution?
You have thousands of data services. So every data service has a particular niche that they are trying to uh trying to support and every time you use a data service you make a copy of data. What happens when you make a copy of data? Yeah, it's a slippery slope. You have your management cost goes up. you're going back into the legacy world where your the multiple data warehouses are cropping up. So what did we do? We started to kind of think about how do we actually simplify our consumption layer.
So we avoid multiple hops. We keep the data in one place and be able to provide the required tooling and access to data in the lake to our consumers. We are at the data bricks conference. So we built the lakehouse on data bricks where we were able to allow secure access to data that was collected in our lake to be able to support a multi-persona uh user base where we're looking at you want to do internal analytics you want to do data bricks uh I'm sorry you want to do data sciences You want to do uh plain old SQL reporting all of it possible through a single uh interface consumption layer.
Of course, all of this fortified by strong governance around security and compliance at a fine grain entitlements level. One of the things that we wanted to also I wanted to also touch upon is we process across 160 different countries. We had to make sure that we uh complied with every local uh law on how we manage the data aligned to their regulations. So we had to come up with various different schemes or very different patterns on how we segregate data uh physical local how do we give access to data and so on.
So we've talked about the foundation. We actually started with the scope. We've talked about the foundation. We've talked about the building blocks. I we went through the problem statement and the solution. Here is a view of our architecture. What did we finally build? Walking through this. So our platform everything is an offering, a component or a product. Walking through this architecture from left to right. There are three main offering product component groups.
On the left hand side is about ingestion and management of data. On the top right is our lakehouse and at the bottom is our API consumption layer. Our injection framework is our engineering centric product that empowers data engineers to create data flows and manage their data on the platform. It is fully cloudnative. It is fault tolerant, testable, uh observable and manages quality through reconciliation schemes and things like that.
It's built on streaming first principles using technologies like Kafka, Delta tables, DLT, data brick SQL, Dynamo DB, Open Search, Kubernetes, Data Dog for monitoring and observability. Our second group of products is the lakehouse which is the number four on the right hand side. Lakehouse is our data sciencecentric product designed to provide self-service access to data and tooling for all our data scientists and analyst to create data science products, BI analytics and plain old simple SQL reporting.
Lakehouse was the first of its kind at JP Morgan built on data bricks again unity catalog data bricks SQL and ML flow it embodies our vision of true innovation at scale by democratizing data and tool while reducing time to market. And our third product group is our API GraphQL based consumption layer which is our developer centric product built for developers to selfserve data through a a GraphQL API. This actually brings our data mesh to life where developers are able to connect to API and get access to in a secure way get access to data and intelligence that is on our platform.
Finally, we wanted to kind of talk to talk about a little bit more about the lakehouse here. We're going to talk about how a data scientist and a data engineer and multiple personas work within our platform. Data scientist. So at the bottom data scientist starts creating their platform uh or their product on a workspace that we've created within data bricks. The data scientist can innovate with the data with secure access to data and having access to the tooling on data bricks such as MLflow and so on.
Most of uh the work that the data scientist does is in an opinionated project skeleton that we have created using uh asset bundles data bricks asset bundles. Once the data break data scientist has actually created their product, they hand it off to data engineers. Now there's a there's a distinction here. A data scientist actually creates the product and the model while the data engineer actually productionizes that model and product is an important distinction because a data engineer then focuses on the scalability and the testability and uh fault tolerance of that product.
The data engineer takes the product through the SDLC life cycle from starting from dev into production. At every point the data scientist al also has access to some of the consumption tools on the workspace and we also have some of the data sharing uh products built on the production side. We're quite excited to have delta sharing um which is a recent addition to our lakehouse. With that I wanted to also uh highlight to all of you for us at JP Morgan democratization of data means inclusion.
It's not just a feature it's a fundamental principle that is embedded in our platform. Thank you. I'm going to have 10 minutes of Q&A.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.