What's a good reference architecture for combining batch and streaming payments data?

Summary

• A unified payment architecture captures both historical batch imports and real-time transaction streams into Delta Lake tables, ensuring that analytics and compliance use consistent data.
• Use Auto Loader or SQL for batch ingestion of periodic settlement data, ledger reconciliation files, and historical records; use Structured Streaming for real-time payment events, fraud detection inputs, and balance updates.
• Both paths write to the same Delta tables with exactly-once processing guarantees, so reconciliation queries can join batch and streaming data without worrying about duplicates or missing transactions.
• Orchestrate batch jobs and streaming pipelines together with Lakeflow Jobs to manage schedules, dependencies, and failure recovery in one place.
• Enable audit trails and compliance checkpoints: Delta Lake's transaction log records every write, and governance tags keep sensitive payment data locked down across queries, notebooks, and BI tools.

What's a good reference architecture for combining batch and streaming payments data?

A unified payment architecture uses Delta Lake as the single system of record for both batch and real-time data. Real-time payment events, transaction logs, and fraud signals stream into Delta tables alongside periodic imports of settlement files, ledger reconciliations, and historical data. Both ingest paths write with exactly-once semantics, so downstream analytics queries can safely join streamed transactions with batch reconciliation files without duplicates. A single orchestration layer (Lakeflow Jobs) runs batch imports on a schedule and keeps streaming pipelines alive, giving your payments team a consolidated view of what data is current, what's been processed, and where failures occurred.

Why separation of batch and streaming causes problems

When batch and streaming data live in separate systems, payment teams face a familiar set of headaches: reconciliation queries need complex logic to decide which system is the source of truth, late-arriving batch records can overwrite streaming data or be missed entirely, and compliance audits require manual cross-reference of two transaction logs. A unified pipeline with a single Delta Lake store eliminates these problems: all writes go to the same tables, all follow the same governance rules, and every action is timestamped in the transaction log.

Key capabilities to evaluate

Capability	Why it matters for payments
Exactly-once delivery	No duplicate charges or missed transactions, even if a pipeline restarts.
Schema evolution	Add new payment fields (new processor IDs, fraud flags, compliance codes) without rewriting history.
ACID transactions	Concurrent batch reconciliation and streaming real-time updates don't corrupt tables; multi-row operations either fully succeed or fully rollback.
Checkpoint recovery	Streaming jobs resume from the last processed offset if a failure occurs, avoiding data loss and missed transactions.
Unified governance	The same access control, audit log, and masking rules apply to batch imports and streaming data.
Cost control	Auto Loader discovers only new files, not rescanning everything; streaming micro-batches batch writes to minimize overhead.

How Databricks approaches unified batch and streaming for payments

Ingestion: Use Auto Loader to efficiently pull batch files (CSV, Parquet, JSON) from cloud storage; it automatically discovers new files and infers schema. For real-time streams, Structured Streaming reads from Kafka, Kinesis, or direct REST APIs and applies transformations row-by-row or in micro-batches. Both feed Delta tables.
Reliability: Delta Lake ensures exactly-once writes by default. Structured Streaming checkpoints store the offset or file position of the last processed record; if a pod restarts, the pipeline resumes without reprocessing or skipping. Batch jobs can be retried with automatic deduplication via Delta's merge operations.
Orchestration: Use Lakeflow Jobs to schedule batch imports (e.g., reconciliation files at 6am UTC) and launch streaming apps as long-lived services. Define dependencies (a reconciliation batch must complete before a nightly settlement report queries the data). Lakeflow captures logs and alerts on failure, so your payments team knows when a pipeline is stuck.
Governance and audit: All writes are logged in Delta's transaction log with timestamps and write-context (which user, which job, which cluster). Tag tables and columns that contain account numbers, transaction amounts, or cardholder data with sensitive-data tags; Unity Catalog enforces row filters and column masks at query time. Compliance queries can always ask who accessed what and when.

Reference architecture diagram

Batch layer: Scheduled jobs (Lakeflow Jobs) read settlement files, chargebacks, and ledger reconciliation from cloud storage via Auto Loader, apply validation and transformation, and write to Delta tables.
Streaming layer: Streaming applications subscribe to payment transaction topics or APIs and ingest events in real-time. Checks for fraud signals, updates running balances, and writes to Delta.
Unified store: Both layers write to the same Delta tables (transactions, accounts, events). Union queries work across batch and streaming data natively.
Consumption: BI tools, reporting jobs, and compliance systems query Delta directly. Lakeflow Jobs orchestrate downstream reconciliation, settlement reports, and fraud alerts.
Governance: Unity Catalog applies access control, masking, and audit logging across all queries and writes.

Common patterns

Reconciliation: Run a batch job that joins settled transactions from a bank file against streamed transaction confirmations. Delta's ACID guarantees let you update the same rows without conflicts.
Backfill and migration: Use Auto Loader's backfill mode to ingest years of historical transactions. Streaming pipelines start once backfill completes, so the system is never vulnerable to double-counting old data.
Late-arriving data: If a payment confirmation arrives late, use a merge operation to upsert the row (insert if new, update if already seen). No duplicates, no data loss.

FAQs

Can I run batch and streaming to the same Delta table?

Yes, both can write to the same table with exactly-once semantics. As long as both use Delta Lake (structured streaming and batch SQL/DataFrames), the system prevents duplicates and preserves order within each partition.

How do I know if data is current or stale?

Delta tables include a _commit_timestamp column in the transaction log. Queries can check when the last write occurred; BI tools can show this metadata. Lakeflow Jobs sends alerts when a batch job or streaming app falls behind.

What if a batch file has the same transactions as a stream?

Use Delta's merge operation: MERGE INTO transactions USING new_data ON transactions.id = new_data.id WHEN MATCHED THEN UPDATE SET ... WHEN NOT MATCHED THEN INSERT .... This upsert prevents duplicates and keeps your source of truth clean.

How much latency should I expect from streaming?

Structured Streaming can process events in real-time (sub-second for simple transformations) or in micro-batches (e.g., every 10 seconds) depending on your configuration. The tradeoff is latency versus operational overhead.