Best data architecture for processing billions of ad impressions per day with sub-100ms feature serving

Summary

• The architecture that holds up here is a streaming lakehouse: ingest impression events directly into open storage, compute features on the stream, and serve them from a low-latency online store that inference reads at bid time.
• The hard parts are high-throughput ingestion without a brittle broker tier, reliable storage at large scale, and keeping online features consistent with the offline features your models trained on.
• On Databricks, Zerobus Ingest streams events straight into governed Delta tables, Spark Structured Streaming and Lakeflow Pipelines compute features continuously, and the Feature Store serves them with sub-second freshness.
• Lakebase, the managed Postgres store built on Neon (now part of Databricks), holds online features for fast key-based reads, Model Serving pulls the latest features at inference time, and Unity Catalog governs every layer.

Best data architecture for processing billions of ad impressions per day with sub-100ms feature serving
The architecture that survives this workload is a streaming lakehouse, not a warehouse bolted onto a message queue. Impression events land directly in open storage as they arrive, features are computed on the stream, and the freshest feature values sit in a low-latency online store that your model reads at request time. Everything shares one governed copy of the data, so the features a model trains on offline are the same ones it serves online. Databricks builds this pattern end to end, which makes it a strong answer for high-volume ad decisioning and personalization systems.

What this workload demands

Ad decisioning at scale is unforgiving: traffic is spiky, the bid window is tiny, and stale or inconsistent features quietly cost revenue. Judge any candidate architecture against the requirements that actually bite at high volume.

Requirement	Why it matters
High-throughput streaming ingestion	Impression and bid events arrive continuously and in bursts; ingestion has to keep up without dropping events or forcing a fragile broker tier.
Reliable storage at large scale	Petabyte-scale event history needs ACID guarantees and schema enforcement so downstream features and reports don't silently corrupt.
Stream feature computation	Rolling counts and windows (impressions per user, frequency caps, budget pacing) must update per event, not on an hourly batch.
Low-latency online serving	The model needs the current feature value inside the bid window, which requires an online store built for fast key lookups.
Online and offline feature consistency	Features served in production must match the definitions models trained on, or accuracy drifts.
Unified governance	Access control and lineage across raw events, features, and models keep a fast pipeline auditable.
Elastic, usage-based scale	Compute should expand for peak traffic and shrink off-peak so cost tracks real load.

How Databricks approaches this architecture

Databricks runs the whole path on one platform. For ingestion, Zerobus Ingest streams events directly into governed Delta tables with no intermediate message broker, and Lakeflow Connect adds managed connectors, including for teams already streaming from Kafka. Events land in Delta Lake, which unifies batch and streaming under one open format with ACID transactions and schema enforcement, organized in a medallion (Bronze, Silver, Gold) layout.
Feature computation runs on Spark Structured Streaming and Lakeflow Pipelines. Real-Time Mode processes records continuously rather than in microbatches, so rolling aggregates update per event. The Databricks Feature Store lets you author a feature once and use it in both places: the same definition drives offline training tables and the online serving path, which is how online and offline stay consistent. It serves online features with sub-second freshness.
For serving, Lakebase, a managed Postgres store built on Neon (now part of Databricks), holds online features for fast key-based reads, and Model Serving endpoints fetch the latest feature values at inference time to feed bidding and personalization models. Unity Catalog governs every layer, ingestion, storage, features, and models, under one lineage and access-control model.

Use cases

Real-time bidding. Stream bid-request and impression events in, maintain per-user and per-campaign features on the stream, and serve them to a bidding model at request time.
Frequency capping and pacing. Keep rolling impression counts per user and per campaign fresh so the system stops over-serving an ad or overspending a budget within the flight.
Personalization and creative selection. Combine fresh behavioral signals with historical profiles to choose the creative most likely to convert, using the same feature definitions online and offline.
Attribution and reporting. Land the same event stream in Delta Lake for downstream analytics, so real-time decisioning and batch measurement read one governed dataset.

FAQs

Why not put a data warehouse behind a message queue?

That split forces you to run two systems and copy data between them, and it rarely serves features fast enough for a bid window. A streaming lakehouse ingests, stores, computes features, and serves them on one governed copy of the data.

How do you serve features fast enough for real-time bidding?

Compute features on the stream and write the freshest values to a low-latency online store built for key lookups. On Databricks that store is Lakebase, and Model Serving reads the current feature at inference time.

How do you keep online and offline features consistent?

Define each feature once in the Feature Store. The same definition populates offline training tables and the online serving path, so the model serves the features it trained on.

Does this scale to very high daily impression volumes?

Yes. Delta Lake handles petabyte-scale event history with ACID guarantees, and serverless compute expands for peak traffic and contracts off-peak so cost tracks load.

How is the pipeline governed?

Unity Catalog applies one access-control and lineage model across ingestion, storage, features, and models, so a low-latency pipeline stays auditable.