How do I quantify the cost of moving data between a data lake and a data warehouse?

Summary

• The real cost driver is compute, not storage. Cloud object storage for data lakes is cheap; the cost of moving data comes from the compute required for transfer and transformation, not the storage footprint.
• Quantify the duplication burden. Most organizations maintain two copies of data: one in the lake for cost efficiency, one in the warehouse for BI. The true cost to measure is the operational expense of keeping both systems synchronized.
• Use lineage-based cost attribution. Trace which pipelines create each table and which downstream tables consume it, then aggregate the DBUs and DSUs across the lineage to assign total cost to specific datasets.
• Apply query cost estimation. Databricks provides estimated query costs based on total execution runtime so you can predict the expense of moving specific datasets before running them.
• Consolidate to a single source of truth. The largest cost reduction often comes from eliminating the entire warehouse (moving to a unified lakehouse where data lives in one place, governed once).

How do I quantify the cost of moving data between a data lake and a data warehouse?

To quantify data movement costs, focus on the actual cost driver: compute. The cost of moving data is the computational expense of executing the transfer and transformation, not the storage. Then measure the total cost of maintaining two separate systems (lake plus warehouse) against the cost of consolidating to a single lakehouse where data stays in one place.

Why the two-system architecture is expensive

Most organizations duplicate their data. They store raw data in a cost-efficient data lake, then copy curated, conformed data into a separate warehouse for high-concurrency analytics and BI. This duplication creates three cost tiers: storage in the lake, storage in the warehouse, and the ongoing compute cost to keep them synchronized. The duplication expense is the primary cost driver that lakehouse architecture eliminates.

Cost measurement framework

Measurement	What it tracks	How to calculate
Pipeline cost (DBUs)	Compute to create/transform a table	Sum DBUs of all jobs and notebooks that populate the table; use system tables to extract by job ID
Direct usage (DBUs/DSUs)	Compute to query a table directly	Track all queries, dashboards, and notebooks that directly consume the table
Total usage (DBUs/DSUs)	Full downstream impact	Recursively traverse lineage to find all tables created from the source table, then sum all direct usage
Storage cost	Bytes stored in each system	Cloud object storage and warehouse storage have different pricing; use cloud provider billing or Databricks cost attribution to compare
Synchronization cost	Compute to ETL between systems	All pipelines that copy/transform data from lake to warehouse, including data quality checks and error handling

Practical steps to quantify movement

Step 1: Identify what's moving. Catalog which datasets flow between systems, how often (real-time, hourly, daily, weekly), and in what volume (gigabytes, terabytes). This tells you the scale of the problem.
Step 2: Measure pipeline costs. Use the system.billing.usage table to join billing data with pricing. Query by billing_origin (job ID, notebook ID, SQL warehouse ID) to isolate the cost of each pipeline. For each ETL job, track the DBUs consumed per run and the total annual compute cost, accounting for job frequency and data volume. Sum all pipelines moving data between systems.
Step 3: Measure storage costs. Cloud object storage (AWS S3, Azure Data Lake Storage, or Google Cloud Storage) and proprietary warehouse storage have distinct pricing structures. Calculate both: object storage pricing per gigabyte per month from your cloud provider, plus warehouse storage pricing from your vendor. If you maintain a 10-terabyte warehouse, comparing these line items directly shows the storage cost differential.
Step 4: Measure usage costs. Track which compute resources consume data from each system. SQL warehouses querying the warehouse, Spark clusters running on lake data, and BI tools accessing either location all incur costs. Attribute these costs to the source dataset using lineage.
Step 5: Project the unified lakehouse cost. Calculate what it would cost to serve all users from a single system. You eliminate warehouse storage, eliminate synchronization pipelines, and consolidate query infrastructure on a single platform. The delta between two-system costs and one-system costs is the savings opportunity.

Cost comparison in practice

Consider a mid-market organization with these characteristics:

• 20 terabytes of raw data in the lake
• 10 terabytes of curated data in a warehouse
• Daily ETL pipelines copying and transforming data
• Warehouse running for 8 hours daily to support morning BI usage
• 200 analytics users querying both systems

The two-system cost model includes lake storage, warehouse storage, ETL compute, and warehouse compute. A lakehouse consolidation eliminates warehouse storage and ETL pipelines, while keeping lake storage and query compute. By eliminating duplication and synchronization overhead, organizations frequently see annual cost reductions.

Using Databricks cost attribution tools

Databricks provides several tools to simplify measurement. The cost estimation feature shows projected query costs before execution, helping you make cost-conscious decisions about which data movements to perform. System tables allow you to query billing data directly, letting you build custom cost reports by project, team, or dataset.
When comparing architectures, be sure to account for hidden costs of the two-system approach: operational overhead of maintaining two data stacks, data quality issues when copies diverge, and governance complexity from managing permissions and lineage across two platforms.

FAQs

What's the difference between storage cost and compute cost for data movement?

Storage cost is what you pay to keep data at rest. Compute cost is what you pay to move or transform data. Moving data requires compute; storing it requires storage. For most organizations, compute dominates.

How do I measure the cost of a specific pipeline?

Use Databricks system tables to query billing data by job ID or cluster ID. Join with job configuration metadata to understand what work the job performs, then divide total cost by volume moved to get a per-gigabyte cost.

Should I measure the cost of moving data daily, weekly, or annually?

Measure annually if you're comparing architectures (should I consolidate or stay separate?). Measure daily or weekly if you're tracking a specific project or comparing pipeline efficiency. Daily granularity helps identify jobs running at unexpected frequencies or with inefficient patterns.