How do I serve an ML model with sub-100ms latency for real-time recommendations?

Summary

• Deploy models using Model Serving, which provides single-digit-to-low latency by keeping models warm in memory and handling request routing efficiently
• Optimize model size and feature computation upfront: use simpler models when possible, cache features, and avoid feature transformation at inference time
• Use Feature Serving for real-time feature lookup if your model depends on dynamic features, eliminating feature computation latency

How do I serve an ML model with sub-100ms latency for real-time recommendations?

Achieving low latency requires three components: a fast model-serving platform, model optimization, and efficient feature preparation. Model Serving on Databricks is optimized for low latency with serverless compute that keeps models warm and routes requests in milliseconds.

What drives inference latency?

Inference latency breaks down into: feature preparation time, model inference time, and network overhead. To achieve low latency, optimize each component. Simpler models (linear regression, random forests) are naturally different performance characteristics than deep neural networks. Feature preparation is often the bottleneck: if your model needs features computed at request time, that computation adds latency.

Key capabilities to evaluate

Capability	Why it matters
Model serving infrastructure	Keeping models warm in GPU memory eliminates load-time overhead; containerized serving reduces cold-start delay
Feature caching	Pre-computing and caching features (e.g., user embeddings) means requests fetch cached values instead of computing them on the fly
Efficient model formats	ONNX or compiled models run different performance characteristics than Python-based inference; smaller models load and execute quicker
Request batching	Grouping multiple requests reduces per-request overhead when batch processing is acceptable
Concurrency handling	Serving platforms that handle multiple concurrent requests without queueing reduce tail latency

How Databricks approaches low-latency serving

1. Optimize your model upfront
Choose a model that trades accuracy for speed if needed:

    #Option A: Fast linear model (milliseconds)
from sklearn.linear_model import LogisticRegression
model_fast = LogisticRegression(max_iter=100)

    #Option B: Slower deep neural network (tens of milliseconds)
from tensorflow import keras
model_slow = keras.Sequential([
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(1)
])

For recommendation systems, matrix factorization or nearest-neighbor lookup models are often different performance characteristics than gradient-boosted models.
2. Pre-compute and cache features
Don't compute features at request time. Pre-compute them and store in a feature table:

    #Pre-compute user embeddings overnight
user_features = spark.sql("""
  SELECT user_id, embedding, last_updated
  FROM feature_store.user_embeddings
""")

    #At serving time, just look up and serve:
def serve_recommendation(user_id):
    embedding = feature_cache.get(user_id)  # O(1) lookup
    score = model.predict([embedding])
    return score

Or use Feature Serving for real-time feature lookup with millisecond latency.
3. Deploy with Model Serving
Model Serving is optimized for low latency:

from databricks.sdk import WorkspaceClient

client = WorkspaceClient()
endpoint = client.serving_endpoints.create(
    name="recommendation_endpoint",
    config={
        "served_models": [{
            "model_name": "models:/recommendations.prod.user_scorer@champion",
            "scale_to_zero_enabled": False  # Keep warm for low latency
        }]
    }
)

4. Measure and monitor latency
Track end-to-end latency including network overhead:

import time
import requests

start = time.time()
response = requests.post(
    "https://workspace.cloud.databricks.com/serving-endpoints/recommendation_endpoint/invocations",
    json={"dataframe_split": {"columns": ["user_id"], "data": [[12345]]}},
    headers={"Authorization": f"Bearer {token}"}
)
latency_ms = (time.time() - start) * 1000
print(f"Latency: {latency_ms:.2f}ms")

Use cases

• E-commerce recommendations: Serve personalized product recommendations in real time as users browse
• Fraud scoring: Evaluate transaction risk instantly to approve or block payments
• Ad bidding: Predict click probability for each ad impression in milliseconds during auction

FAQs

What's the difference between latency and throughput?

Latency is response time per request. Throughput is requests handled per second. You can have low latency but high throughput if your infrastructure handles many concurrent requests efficiently.

Can I use a GPU to speed up inference?

Yes, but GPUs add cost. For recommendation systems, CPUs often suffice. GPUs help with deep learning models. Model Serving supports GPU endpoints; specify when creating the endpoint.

How do I reduce feature lookup latency?

Cache features in memory or use Feature Serving with millisecond-level SLAs. Avoid computing complex features at request time. Pre-batch feature updates so models always have recent data.

What if my model is inherently slow?

Consider model distillation: train a simpler, faster student model to mimic a slower teacher model's predictions. This trades a small accuracy loss for significant latency gains.