How do I combine historical and streaming data for real-time predictions?

Summary

• Define each feature once so the same logic computes over historical batch data and live streams, then serve those features from a low-latency store at prediction time.
• The failure to avoid is train/serve skew: the model trains on one version of a feature but sees a different value in production. Point-in-time joins fix it.
• Keep an offline copy of features in Delta for training and batch scoring and an online copy for real-time lookups, both from one governed definition.
• On Databricks, Feature Engineering in Unity Catalog, Spark Structured Streaming, online feature stores backed by Lakebase, and Model Serving cover the batch-plus-stream-to-real-time path end to end.
• MLflow registers the model in Unity Catalog and marks the live version, and a served model can look up its features automatically from the online store using just the primary key.

How do I combine historical and streaming data for real-time predictions?

Define your features once, compute them over both historical batch data and live streams, and serve them from a low-latency online store so the model scores on fresh values the moment a request arrives. The hard part isn't running a model in real time; it's making sure the features the model sees in production match the ones it trained on. You solve that with a feature store that keeps an offline copy of each feature for training and an online copy for serving, plus point-in-time joins that reconstruct feature values as they were when each historical label was recorded.

The core problem: train/serve skew

A real-time model learns from historical data but predicts on live data, and the two paths often compute features differently, so the model degrades in production even though it looked accurate in training. That gap is train/serve skew. Closing it means the same feature definition runs in both places, historical training rows reflect the feature's value at the moment of each label rather than today's value, and the online store returns the current value at inference. Then adding a streaming source to a model that already uses batch features is a small change, not a rebuild.

What to get right

Capability	Why it matters
One feature definition for batch and stream	Reusing the same logic for historical and live data is what prevents skew and duplicate pipelines.
Offline store for training and batch scoring	A durable, queryable history of features lets you train and backfill from one source.
Online store for low-latency serving	Real-time predictions need fast key-based lookups, not a scan of historical tables.
Point-in-time correctness	Training rows must reflect feature values as of each label's timestamp to avoid leakage and skew.
Automatic feature lookup at inference	The serving layer should fetch features by key so the caller sends an ID, not precomputed features.
Governance and lineage across data, features, and models	One permission and lineage model over the whole path keeps real-time systems auditable.

How Databricks approaches it

Feature Engineering in Unity Catalog lets you define a feature once and use it everywhere. Features become governed Unity Catalog objects, materialized as Delta feature tables in the offline store for training and batch inference. The same definition can read from a batch source or a streaming source, so moving a feature from batch to streaming is a code change rather than a second pipeline.
For streaming features, Spark Structured Streaming computes values from the live source and writes them to the feature table, while Databricks keeps an offline copy of the ingested data. When you build a training set, the feature store computes the same values it would have produced for historical timestamps and does point-in-time joins against your labels, so training reflects what was known at each point in time. Batch and streaming features can sit in the same training set and the same model.
At serving time, features come from an online store. Databricks online feature stores are backed by Lakebase, the Postgres engine (built on Neon, now part of Databricks) that gives real-time inference fast key-based lookups. You sync from the offline Delta tables using publish modes: triggered for incremental syncs, continuous to update the online store as new data lands, and snapshot for a full copy. Feature Serving can also expose features directly to applications without a model.
Models tie it together. A model logged with Feature Engineering keeps references to its features, so at inference Model Serving locates the online store and retrieves features automatically: the caller passes a primary key and the endpoint assembles the rest. MLflow manages the lifecycle. Register the model in Unity Catalog and pin the live version with a @prod alias, with @challenger for A/B evaluation, so you always know which version is serving and what it trained on.

Use cases

Fraud detection. Combine long-run account history with features computed on the live transaction stream, then score each transaction as it arrives so risky activity is caught while it's still actionable.
Real-time personalization. Blend historical preferences with in-session clickstream features to rank the next recommendation on current behavior.
Predictive operations. Merge historical baselines with streaming sensor or usage data to flag anomalies and forecast demand or failures.

FAQs

How do I stop my real-time model from drifting between training and serving?

Use point-in-time joins so training rows reflect each feature's value at the label's timestamp, and serve the same feature definitions from an online store at inference. That consistency removes train/serve skew.

Can one feature definition handle both batch and streaming?

Yes. In Feature Engineering in Unity Catalog the same feature can read from a batch or streaming source, so historical and live data run through identical logic and land in one feature table.

Where do features come from at prediction time?

From an online store backed by Lakebase, which serves fast key-based lookups. A model logged with its feature references has Model Serving fetch those features automatically, so the request only needs a primary key.

Is this generally available on Databricks?

Feature Engineering in Unity Catalog, streaming features, point-in-time joins, online feature stores, Feature Serving, and Model Serving are generally available. The newer declarative Feature Views framework is in Public Preview.