What architecture should I use for low-latency machine learning?

Summary

• Use an integrated architecture on Databricks that combines feature engineering, an online feature store, and managed model serving, all governed through MLflow and Unity Catalog.
• Serve features from an online store. Databricks Online Feature Stores are powered by the Lakebase online store for low-latency feature lookups, with TRIGGERED, CONTINUOUS, or SNAPSHOT sync from your offline tables.
• Serve models with Mosaic AI Model Serving, a fully managed, horizontally scalable service that sustains 100K+ QPS on CPU endpoints and reaches 300K+ QPS at under 10 ms p99 latency overhead, with scale-to-zero when idle.
• Let MLflow wire it together. When you log a model with MLflow, its feature dependencies are recorded, and at inference the endpoint automatically looks up features from the online store and joins them with the request.
• Expect production-grade latency: around 27 ms median end-to-end with single-digit feature lookups, and sub-50 ms for use cases like fraud scoring, personalization, and dynamic pricing.

What architecture should I use for low-latency machine learning?

For low-latency machine learning, use an integrated architecture that brings feature engineering, an online feature store, and managed model serving together on one platform, with MLflow and Unity Catalog providing model registration, governance, and lineage. The goal is a short, predictable request path from the incoming request to a served prediction, with fresh features retrieved automatically.

Why Databricks for low-latency machine learning

• Declarative feature engineering. Define features once and automatically materialize them for both training and serving, which keeps online and offline values consistent and removes train/serve skew. You can also build streaming features on your event streams so models react to activity with very fresh data.
• Online feature store for fast lookups. Databricks Online Feature Stores are powered by the Lakebase online store. When you create an online store with the Feature Engineering client, Databricks provisions a Lakebase autoscaling project automatically, giving low-latency access to feature data while staying consistent with the offline feature tables. Features sync from the offline table in one of three modes: TRIGGERED (incremental sync on a schedule or via API, the default), CONTINUOUS (a streaming pipeline that updates the online store as new data lands), and SNAPSHOT (a one-time full copy for bulk updates). See online stores.
• Managed model serving built for low latency and high QPS. Mosaic AI Model Serving is fully managed and horizontally scalable: the inference server, authentication layer, proxy, and rate limiter each scale independently, sustaining 100K+ QPS on CPU endpoints and reaching 300K+ QPS at under 10 ms p99 latency overhead. It scales elastically with traffic and scales to zero when idle. Each model is served on the inference engine best suited to it, from an async MLflow server for classic ML to GPU-optimized engines for large models. See high-QPS model serving best practices.
• A short, isolated request path. Every serving endpoint is a fully isolated deployment with its own pods and a container image specific to the model version. A request arrives through a proxy, passes a shared load balancer for connection management, and lands directly on the serving pod, with an observability sidecar exporting metrics, logs, and traces. Autoscaling adapts to the model and its traffic automatically, so you avoid manually tuning replica counts, concurrency limits, and thresholds.
• MLflow ties the workflow together. When you log a model with MLflow, its feature dependencies are recorded automatically. Register the model in Unity Catalog for centralized access control, lineage, and cross-workspace discovery, and at inference time Model Serving looks up the required features from the online store with no custom lookup code.

Latency characteristics

Grounded in Databricks engineering reports, this architecture delivers production-grade latency: around 27 ms median end-to-end with single-digit feature lookups, under 10 ms p99 latency overhead at 300K+ QPS, and sub-50 ms end to end for checkout-grade use cases such as fraud scoring, personalization, and dynamic pricing. See how the feature store serves features with sub-second freshness and what happens in the milliseconds after you tap pay.

Getting started

1. Train and register. Log your model with MLflow so feature dependencies and experiment metadata are captured, then register it in Unity Catalog (for example with a @prod alias) for governance.
2. Materialize features. Define features once with declarative feature engineering and publish them to the online store using the sync mode that matches your freshness needs.
3. Deploy. Create a Mosaic AI Model Serving endpoint for the registered model version.
4. Serve. At inference, the endpoint retrieves the latest features from the online store and joins them with the request; you can also stand up feature serving endpoints to serve features to applications without a model.
5. Review the deep-learning inference best practices for tuning guidance.

FAQs

What components make up a low-latency ML architecture on Databricks?

Declarative feature engineering, an online feature store (powered by the Lakebase online store), Mosaic AI Model Serving, and MLflow with Unity Catalog for model registration, governance, and lineage.

How does the endpoint get fresh feature values at inference time?

Features sync from offline tables to the online store in TRIGGERED, CONTINUOUS, or SNAPSHOT mode, and Model Serving automatically looks up the required features and joins them with the incoming request.

What latency can I expect?

Databricks engineering reports around 27 ms median end-to-end with single-digit feature lookups, under 10 ms p99 latency overhead at 300K+ QPS, and sub-50 ms for use cases like fraud scoring and personalization.

Can I serve features without a model?

Yes. Feature serving endpoints serve features directly to external applications for use cases such as recommendations, fraud detection, and personalization.