Building a Feature Platform with Databricks: JPMorgan Chase Case Study

Summary

JPMorgan Chase built a unified ML feature platform on Databricks to serve 80 million US customers, solving three critical problems: feature duplication across teams, training-serving skew from inconsistent computation, and compliance blind spots from untracked pipelines.
The platform centralizes feature engineering, provides consistent offline and online feature stores, and delivers end-to-end lineage from raw source data through model training and inference using Databricks Structured Streaming and Unity Catalog.
By unifying feature development on Databricks, JPMorgan Chase accelerated ML development from months to days while embedding governance controls, access management, and feature drift monitoring at every stage of the pipeline.

Building a Feature Platform with Databricks: JPMorgan Chase Case Study

Building a scalable feature platform for 80 million customers requires unified governance, preventing duplicate feature development and ensuring consistent model training and serving. JPMorgan Chase's enterprise ML teams faced three critical problems: feature duplication across teams, training-serving skew from different computation methods, and compliance blind spots from untracked pipelines.
This talk explores how JPMorgan Chase built a unified ML feature platform on Databricks that eliminates duplication through centralized feature engineering, provides consistent feature serving via offline and online feature stores, and enables full end-to-end lineage from source data through training and inference. The platform leverages Databricks structured streaming, feature pipelines, and Unity Catalog to accelerate development from months to days while embedding governance controls, access management, and feature drift monitoring at every stage.
🤝

Chapters

FAQs

What problems does a unified feature platform solve for a large bank like JPMorgan Chase?

Without a shared platform, different ML teams independently build the same features, wasting engineering effort and creating inconsistencies. JPMorgan Chase also faced training-serving skew, where features computed differently at training versus inference time degraded model accuracy, and compliance gaps from pipelines that were not tracked or audited.

How does JPMorgan Chase prevent training-serving skew in its ML models?

The unified feature platform ensures the same feature definitions and computation logic are used for both offline training data preparation and online inference-time serving. This consistency eliminates the skew that occurs when teams compute features differently in different environments.

What role does Unity Catalog play in JPMorgan Chase's feature platform?

Unity Catalog provides governance, access management, and end-to-end lineage across the feature platform. It enables the bank to track which features are used in which models, enforce access controls on sensitive features, and meet compliance requirements by maintaining an auditable record of the data pipeline.

How does the JPMorgan Chase feature platform support generative AI use cases?

The video discusses how a centralized feature store enables generative AI use cases through feature reuse — features developed for traditional ML models can be surfaced and reused by generative AI applications, accelerating development and ensuring consistency across model types.

Full transcript

[00:08] Good afternoon, everybody. Uh I'm Lohit Giri. Uh I'm senior director of AIML architecture and generative AI platforms at JPMC. Uh I'm accompanied by my principal architect, Hong Ji, and the product lead, Terence Witcher. We're trying to
[00:24] talk about the feature platform that we built at JP Morgan Chase uh last year or so, taking database as an example. If there is a mistake in the slide, if you guys know features, the slides are prepared by generative AI, so there
[00:40] could be a uh some skew problem in the data there, so that could be the problem, right? step back. What's the feature platform? I'm going to talk about why we build a feature platform.
[00:56] The the reason to build a feature platform. Uh you go to your holiday. You tap your card at a coffee shop, right? While you tap your card, the card swipe happens, you get the
[01:11] coffee. In between that transaction that happening is there are multiple things happening in the back end, right? The card provider is finding your location. They know what what you have been doing. Are you a trusted person?
[01:27] Then they know where you are. Uh the location of the place where you're swiping in, and then they're trying to find out uh is it a fraud happening or not happening? And there are definitely a dozen different calculations that's happening in the real time. And while I spoke all this, it need to happen in
[01:43] almost like a flip of a second, right? So, that's is powered by a model. The model need data, and that data is the feature, right? The feature is a calculation, a numerical value, a fraction value, or the location ID that has been sent sent
[02:01] to the model, right? This is what the feature is, right? So, it's very important to know about the feature, right? Like, why do we build the feature? The The moment we move into uh the application of feature platform, think of it for a bank of our size, JPMC,
[02:18] this is just one use case. We have around 80 million US customers. We are running 80 million US customer those transactions swipes are happening. You have to run them at scale. You have to run those at different vendors. And 80 million people across the world they are traveling, right? So,
[02:34] you have to do all at very very very large large scale, right? And have to be accurate about it because otherwise there could be fraud happening. There could be a refund that people going to ask for, "Hey, I got charged double. I have to get it back." So, there's more customer churn that come back to it, right? All of this is uh powered by
[02:50] feature calculated in the back end, right? So, that's what we need to do. Uh So, the subject of talk here is feature platform uh powering our machine learning ecosystem uh overall. So, uh Hong will talk about it more detail. Uh
[03:05] when we build the feature platform, multiple teams build their own product thinking, but then there are three major problems that they run into, right? So, for the same example I told you earlier, there's a fraud team building the model about
[03:21] customer spend last 1 month. 2 months later the marketing team is looking for the same thing, a customer spend last 1 month. And then the we have a marketing team that is also building a model, customer spend last 1 month. So, they're building three different
[03:37] dimensions about the feature, right? So, this leads into a duplication, right? So, duplication is another problem. So, uh you are This is for people executives, you have three teams working on same thing. You are spending money on the compute, right? So, that is very
[03:52] important to use feature reuse. And then it's also governance headache, right? So, because you need to make sure all the features are calculated from one trusted store. There's auditability around it, right? So, that's that's where the feature platform is. So, that's the first one, the duplicate duplicate feature. The second one is
[04:09] training and service queue. Right? When features computed differently and served differently and their different model techniques are determined, so they can degrade. There is not a central way to provide the feature uh served in production. Right? So, that is
[04:25] very hard to provide monitoring on top of it, right? That's the second dimension that is the problem that comes in, right? The last thing is I already talked about the compliance blind spot, right? Three different pipelines, three different engineering teams all running them together in feature and it becomes just tedious. So, there's no lineage,
[04:41] there's no tracking, there's no reuse. So, how the how we going to find control over it, right? So, these are the three different dimension problems that we run in. Over the time today, 30 plus minutes, we going to talk about what we're building, a unified platform
[04:57] for governing the features, right? So, this platform will uh will talk about the budgeting problem, the data serving problem, data discovery problem, feature calculation problems, and how do we store it and serve it and make it reusable. So, that is the foundation of
[05:13] this. We are building it as a unified platform that serves across all line of businesses and powering all our mammal. So, I'm going to hand over to uh Hong and Hong will talk about the bird's-eye view, the architecture of it, how we have
[05:28] delivered it. So, over to you, Hong. So, you circle when I uh doing the we're doing the machine learning research.
[05:48] And machine learning So, it's easy to when I have data, right? So, I can do data exploration and uh feature engineering and uh and try different algorithm and then come up with some machine learning model
[06:04] and then test and then everything oh, great. But, the most time uh the structure time spent on is uh get to the data. Very difficult. I believe most of people here work with either to research or work with your
[06:22] machine learning teams. Uh most of time spent on is uh get to the data ca- capture the data into uh right environment for your data scientist to experiment. So, uh we build a production experimentation
[06:38] environment. We believe data scientist most efficient for data scientist to come up a model using using uh available product uh product data. Yeah. I will speak a little bit louder so
[06:54] product data available to them. So, uh batch data uh stream data and so, they can uh in a environment they connect to those sources ready for them to connect.
[07:10] All we need to just uh onboard the team into the marketing environment. And then each team uh with uh right access granted to them and then they can start to explore uh
[07:26] usually you can use a Jupiter notebook for example, explore data engineer data as a batch and then try different and then select uh select the right algorithm, test out, and come up with
[07:41] uh right models and then even writing code and then do model inference. And then this pipeline will be available and then test out and then check out into a like a bit bit of
[07:59] buckets and then hand over to engineer team. So through the STLC life cycle. So there will be three different type of pipeline on here. Feature engineering, engineer features from source
[08:14] and training model. So based on features and then train the model. And the next last one is a serving. Here we'll be have a production ready serving code available to
[08:31] deploy the batch or deploy the in a real time environment and then to for batch inference or serving serving event. So through this process the key point is we need that environment
[08:48] to govern the data access. So where's the source data come from? Where's the features engineering into? Where the key point is a feature store. The feature store will be most of the will be offline feature
[09:05] store as a batch. So this will be available for batch to train the models and also for batch inference. And offline feature store, the features available there can be power online
[09:22] feature store sinking to there or you can engineer features from from a streaming process into online feature store for the low latency freshness feature freshness.
[09:37] So, the features will be available for real-time serving. So, depend on the your real-time uh latency requirement, could be under 100 milliseconds, could be maybe 50
[09:55] milliseconds. This is very critical to for online feature store. Can be powered in a different system for example. Uh Yeah, you guys already attend the such a keynote today.
[10:12] Uh Databricks have a database. And in other environments, you can use a Postgres for example. I believe some people using uh like Redis for example for the online feature
[10:28] store. And uh From here here, then we The features are available in the offline feature store and the online feature store. So, when the um uh when we do the real-time serving
[10:45] or batch batch inference, how we construct the feature vectors in batch or in real time for the real-time serving or batch inference. So, here is a key to
[11:01] uh we need to unify the way to manage the the access control to different application to get the features in a batch or in a uh in a real-time inference. So,
[11:18] there's maybe you may have a hundreds of different uh machine learning teams create different use cases, different models. A lot of features common. The people may I I do the fraud, for example. I have
[11:34] calculate the uh average of the prices or spending. Another use case may doing a similar. They end like repeatedly this calculation. So feature store we allow encourage
[11:50] people to computer once, reuse many times. So that's what uh feature store so feature the power of the feature store here. And the lineage.
[12:07] So we have a source data. We have features engineered. We have feature store in uh feature store. And then we have model trained and uh in the model repo. We register this model and then we have
[12:24] same model could be uh leveraging the different environment batch inference or real-time inference or maybe use case one, use case two for your inference, right? So how I ensure end end to end uh lineage. So source data,
[12:42] feature engineering pipelines, and uh training pipelines, features training pipelines, and the models, and the model inference different inference and the inference output. This is the key to for a lot of time we have a problem say
[13:01] where's the problem coming from? How do we troubleshoot this problem? So if we have a clear clean uh lineage captured end to end. So this will uh first will have a uh our teams to troubleshoot those
[13:18] problems. Where with a graph you can immediately to see where's the point of which node which is uh what what the root cause could be happening over there. And the second is a
[13:35] What second is a We with the link with the lineage we will be able to uh check in the different versions of that data or features or models used in for right application, okay.
[14:01] So Hong took you through the architecture in the in the solution. I'm going to frame this more in terms of the the tenants that we have on the platform. So as you onboard to the platform, now you have access to all your data, right? So we already have all the governance controls embedded into the feature platform, right into the experimentation
[14:18] platform for your model development. Um so the the onboarding path here is really fast. You have the tools available to you, so you're familiar hopefully now with uh Databricks. Um to be able to use those tools to experiment, build features, start to develop a a model. Um and now we can
[14:35] then provide this pathway. So Lowhit mentioned the SDLC pathway, so CICD path into production. So you've developed your features, right? Automated pathway into productionizing those those features, right? So being able to go from zero to production, right? In in
[14:53] days. Um where you know, previously it took months. Um model training. So now that you have your features computed into production, right? Backfilled with all that history, we can do that authoritative training that we referenced before, right? With all the
[15:09] versioning, all the lineage, all the traceability, right? From now for that model artifact back to the source systems, right? Um and then same thing with serving, right? Take it one step further. Now we're generating those inferences, we're writing them to
[15:24] the lake, we're writing them to our our attribute to to events, to a hub. And we have that full lineage with the versioning of the features, right? And we're able to do this for each of our tenants, right? So, the point here is, you know, we're doing this for an
[15:40] enterprise, but then we think about each of the tenants that we have in our platform, each of the sub LOBs that we support within Chase. Right? And these benefits start to compound. Right? So, you know, not just accelerating one team, right? But how do we accelerate How do we accelerate many lines of
[15:56] businesses across the business? Uh just to double click, I mentioned these, but traceability, like it was We built this to solve two things. Like, one was the
[16:11] business problem, right? How do we accelerate time to value? Right? But how then how do we do that by embedding controls? And it wasn't because so we we we know. We need to embed controls, but one of the ways we get to acceleration is embedding the controls into the
[16:26] platform, right? Making sure we have the traceability. It's going to help you with your development, right? But it's also going to help, right, when you when you need to do um you know, at least in the financial services like, you know, make sure we we're we're audit ready, right? It helps accelerate us, right? You know, from the
[16:43] development standpoint, but also just managing the life cycle of that feature of that model. Um so, we have the traceability, we have the lineage. We have the access controls built in, right? So, um we have access controls at a um
[16:58] at a feature level, right? And so, we can make sure that um only the the teams, the systems that are supposed to have access to that feature do have access to that feature. And then we have the operational visibility. So, we have the monitoring built in, uh monitoring
[17:15] in terms of the the platform, yes, but for the the drift of those features, right? So, if you think about you've trained a model, right? And now you're serving it. Like, how is that the the features that you're using for the serving, how do they differ, right, statistically from what you trained it
[17:30] on, right? How are you able to do that at scale? How are you able to do that consistently? All right, so we talked a lot about traditional ML, so that's that box on the right, sorry.
[17:46] Um Now, we're also thinking about now, how do we How do we extend the use of these features that we have? If you think about it in the generative AI stand, the generative AI view, uh you can take a feature, maybe that you're using for personalization, right? You're using it
[18:02] in your ML model to, I don't know, to to personalize your your mobile app experience. You can take that same feature, incorporate it into your prompts to help drive that same level of personalization, right? And it also drive consistency,
[18:19] right? So, now you're improving the quality of your your prompts, you're re-le- reusing that same feature, right? Uh and you're also providing consistency in like different experiences in your um in your applications.
[18:36] Hong was talking about the the embedding pipelines, right? How to leverage feature store for embeddings. Um you know, something you're also going to be leveraging in your agent AI applications. And then hybrid uh retrieval. So, here again, like the the power of leveraging your online store. So, if you're doing you know, a real-time rag,
[18:53] um you can incorporate, you know, multiple sources in hybrid the hybrid here like bringing the embeddings, bringing that structured data, right? Incorporating that into uh your prompts. Awesome. All right. Okay, thank you. Thank you, everybody.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.