Get Data Into Databricks - Feature Store

Summary

• Feature engineering automation: Transform raw data through joins, aggregates, and transformations to create reusable features for machine learning workflows.
• Offline and online serving: Use feature tables for offline batch processing during model training and online real-time inference serving from one store.
• Consistent feature usage: Feature tables ensure consistency between offline and online serving, addressing the online-offline skew problem in production ML.

Get Data Into Databricks - Feature Store

Try Databricks today: https://dbricks.co/3EAWLK6.
Link to the code: https://www.databricks.com/resources/demos/tutorials/data-science-and-ai/feature-store-and-online-inference
Discover Databricks Feature Stores in machine learning workflows. We will walk through how raw data is transformed through feature engineering processes like joins, aggregates, and transformations. These features are then stored and made discoverable for reuse in both model training and serving stages. The system ensures consistency between offline batch processing and real-time online serving for inference. A searchable interface allows users to create, discover, and manage features efficiently, enabling seamless integration into client applications.

Chapters

• 0:00 Introduction to feature stores
• 1:05 Creating a feature table
• 2:39 Building training data
• 3:30 Training and logging the model
• 4:21 Real-time inference

FAQs

What is a feature store in Databricks?

A feature store is a system for creating, discovering, and reusing features in machine learning workflows. Any Delta table with a primary key becomes a feature table once you specify the primary key.

What is the difference between offline and online feature serving?

Offline serving uses feature tables for batch model training and inference, while online serving makes features available for real-time predictions in production applications.

Do I need a feature store for production machine learning?

This video introduces feature stores as a core component of production ML on the Databricks Data and AI platform. A feature store—implemented as any Delta table with a primary key—creates, discovers, and reuses features across your organization. By defining primary keys on transformed data, you gain the ability to serve features both offline (for training) and online (for inference) through a unified system.

How do I prevent training-serving skew?

This video demonstrates how the Databricks feature store prevents training-serving skew by using a single Delta table as the source of truth for both training and online serving. Once you define a feature table with a primary key, the same features are accessible for model training and real-time inference, ensuring consistency between training and production data pipelines and eliminating data drift issues.

Which platforms support real-time feature computation and online serving?

This video shows how the Databricks feature store supports both real-time feature computation and online serving from a single Delta table. The platform enables you to serve feature tables online for low-latency inference while maintaining consistency with offline features used for training. This unified approach eliminates the complexity of managing separate feature systems and reduces infrastructure overhead.

Full transcript

0:00 Introduction to feature stores

Hi everyone in this video we'll be talking about the feature store in data bricks generally a feature store is used to create discover and reuse features in data bricks a feature tape was any Delta table with a primary key that means that when you featuri your raw data by applying any number of joins Aggregates and transforms all you have to do to that table to make it a feature table is ify a primary key at this point there are a couple of different paths you can serve that feature table online for realtime feature suring this is known as an online table or or an online store too you can also use it in an offline or a batch setting for model training and inference when you use a feature table throughout the model life cycle you ensure consistent usage of features during model training for which addresses a key component of online offline skew in this notebook I'll go over how we can do that in data breaks this example we are a travel agency that wants to provide recommendations to our users uh to do that we need a model that predicts whether or not it's likely that a given user is going to book a um to make a booking or not we have some data on our users will apply some transformation logic to convert it into the features that we'll then use to train our model to do that we'll just select the data that we have specify and apply the transformation logic don't worry about the details and store the results in a data frame this point to turn that table into a feature table we can use the feature engineering client which is a comprehensive API to create manage and use features to create the table itself I just specify the name of the table the primary key the data itself along with the metadata that I want to have associated with it this feature table is now ready and directly integrated and governed by unity catalog we look at a high level overview of the data we can see that the ID is a primary key and walking over to the lineage tab we can see from the lineage graph that this feature table is Downstream of the original data and Upstream of the model that will train with our feature table we can create the training data set by specifying the ID which we'll use to um in the feature lookup to look up the features associated with that ID combine it with our purchase label and then use it to train our model so to train the data set grabbing the ID and the label I'll use feature lookup to use those IDs to look up this list of features from the feature table that we just created and then to create the actual trading data set itself I'll combine the original data with the features that I just grabbed minus sum specify that the label is the purchase column and I have my training data set next we'll train the model the details of how uh what model we train aren't important for the Intensive purposes of highlighting the benefits of using a feature table throughout this model life cycle what is important is that when you log a model that's been trained with a feature table using the client uh feature engineering client one of the things that's stored along with the metal met the model metadata is the feature spec the feature spec specifies the inputs to the model along with how to get them for example the destination ID is an input to the model and it's retrieved with the lookup key ID so when we have our model and have moved it to production we can run inference on it to do that all we need to specify is the ID on the back in there's automatic feature lookup that takes this ID grabs the features that are associated with that ID uses those same features as the inference features this is what ensures consistent usage of um features during model training and inference and then forecast whether a user is likely to make a booking or not here in this example we'll just select the IDS that we had in our table this is contrive in a production setting you might have a data bricks job that updates the feature tables with new user ID information run inference we just feed the IDS to forecast and we get the prediction so in this demo I showed how we can create and use a feature table throughout the model life cycle you can find this notebook um along with others on deeper dive topics such as point in time lookup which involves on thefly feature calculations deploying online tables as well as streaming feature tables at DB demos.

4:21 Real-time inference

There you'll also find other helpful tutorials that span the entire data breaks platform from data engineering data science and AI data warehousing and bi along with data coverance thank thanks for watching

Learn more about the Databricks Data and AI platform.