Cloud Cost Optimization: Adobe's Real-Time Databricks Monitoring

Summary

Adobe's digital imaging team built a three-tier real-time cost monitoring system on Databricks system tables with zero new infrastructure to close two gaps: an attribution gap where SQL warehouse costs were assigned to creators rather than query executors, and a latency gap where billing data surfaced days or weeks after spend occurred.
The three tiers address early warning alerts via REST API monitors, authoritative billing attribution, and query-level cost attribution that traces spend to the individual analyst or data scientist who ran each query.
Implementing the system reduced SQL warehouse costs by 51% in 30 days, serverless compute costs by 14%, and policy violations by 57%, providing a blueprint other teams can implement without additional tooling.

Cloud Cost Optimization: Adobe's Real-Time Databricks Monitoring

Cloud costs are rising faster than budgets, and most teams discover expensive operations only after spend has occurred. At Adobe, the challenge was acute: SQL warehouse was rolled out across hundreds of data scientists, but costs were assigned to warehouse creators rather than query executors, creating blind spots on millions of dollars. Latency made it worse, with billing data surfacing weeks later and cost decisions already locked in.
Rajeshwari Raghuraman, Senior Manager of Data Science Engineering at Adobe, walks through a three-tier architecture for real-time cost monitoring built on Databricks system tables with zero new infrastructure. Learn to implement REST API monitors for early warning alerts, billing attribution for authoritative spend data, and query-level attribution to close the creator-executor gap. Discover how Adobe reduced SQL warehouse costs by 51% in 30 days, serverless compute by 14%, and violations by 57%, while building modular, configurable monitoring that scales from hundreds to millions in annual spend.

Chapters

FAQs

What was the attribution gap that Adobe needed to solve in its Databricks cost monitoring?

When SQL warehouse was rolled out across Adobe's data science team, the billing system attributed costs to the warehouse creator rather than to the person who ran each query. This created a blind spot on millions of dollars of spend that had no traceable source in the existing cloud operations dashboard.

How does Adobe's three-tier monitoring architecture work?

The first tier uses REST API monitors to generate real-time alerts before costs accumulate. The second tier consumes Databricks billing data to provide authoritative spend attribution. The third tier adds query-level attribution that resolves the creator-executor gap, linking each query's cost to the individual who ran it. This video explains that all three tiers must work together—fixing one gap without the other still leaves teams flying blind.

What results did Adobe achieve after implementing real-time cost monitoring?

Adobe reduced SQL warehouse costs by 51% within 30 days, reduced serverless compute costs by 14%, and reduced policy violations by 57%. The Senior Manager of Data Science Engineering presenting this video describes these outcomes as achievable through a modular, configurable design that scales from hundreds to millions of dollars in annual spend.

How did Adobe build cost monitoring with zero new infrastructure?

Adobe's monitoring system is built entirely on Databricks system tables, which capture billing and query metadata natively within the platform. This video explains that by using existing system tables rather than external tooling, the team avoided provisioning additional infrastructure and could deploy and iterate on the monitoring system quickly.

Full transcript

[00:08] Welcome and thank you for joining me today on Adobe's inside Adobe's near real-time cloud monitoring system. Let me start with a question. When was the last time you discovered an expensive query was running? Raise your hands if your answer is after
[00:25] the cost has been already incurred. Ooh. A lot of them. I think I should have raised my hand, too. You're not alone. Over the next 15 minutes or so, I will be covering four things. The problem we were facing, how we built
[00:41] a three-tier solution, and detailed walk-through of each of the solution, and how it runs in production today, and what we got out of out of it. So, by the end, you will have a blueprint you can take and start executing on your Monday morning.
[00:56] Does that sound exciting? Cool. I'm Raghu Raman, and I'm a senior data science engineering manager in Adobe's digital imaging team.
[01:12] Let's start with the problem. So, when I joined the digital imaging team, managing cloud budget was one of my responsibilities. The cloud ops dashboard from the platform team was my starting point. Good high-level visibility. It served its purpose
[01:27] until the questions got harder. Around the middle of 2025, SQL warehouse was rolled out across the data science team. Suddenly, every data scientist and data analyst could run queries without spinning up an ad-hoc cluster.
[01:44] Frictionless access, great for productivity, isn't it? But, I had accountability to finance and executive leadership on the cost budget. So, the cost are already climbing fast,
[01:59] especially for the SQL warehouse. And billing system attributed cost to the warehouse creator, and not the executor. And I needed query level attribution, and that gap was mine to close. That's what the system is built to fix.
[02:18] There are two distinct gaps, and both had to be solved together. The attribution gap. Billing assigns SQL warehouse cost to the warehouse creator, not the person who ran the query. So, we had spent accumulating with no way to try trace to its original source
[02:35] using the CloudOps dashboard. A blind spot on millions of dollars. Next, the latency gap. By the time visibility surfaced, the cost was already incurred. No signal when the query was running. You would find out days later, maybe
[02:51] sometimes weeks, occasionally only when finance being due, "Hey, why did I see a $6,000 bill?" So, fix one without the other, and you're still flying blind. So, you either know who drove the cost, but too late, or you see the spike, but
[03:07] can't attribute it. That's why this had to be one system solving both simultaneously. The good news, we didn't need to build any new infrastructure.
[03:23] Every signal we needed was already sitting in the native Databricks system tables. The REST API tells you what's running right now, live queries, seconds of latency. Query history tells you when a warehouse is actually doing work versus just
[03:38] idling. Billing usage gives you authoritative DBU usage per billing interval. Billing list prices has cost per SKU name. The architecture is about connecting the dots that were already there. The signals existed. We just needed to
[03:54] join them. Three tiers, inside each solving a different problem. None of them replaces the other. Tier one is a REST API monitor. It catches long-running queries in near
[04:09] real time while they are still running with seconds of latency at a low cost. Think of this as your early warning system. Tier two is a billing track. This is where you get authoritative dollars and DBUs pulled from billing usage and list prices. That's about
[04:26] approximately 2 and 1/2 hours of log, but this is the ground truth of what was actually spent. Tier three is a SQL warehouse attribution, and this is where we close the creator and executor gap. We join warehouse events, query history, and billing usage to get the cost at the
[04:44] query level per user. Now, two data points run through the system. Tier one stands alone and alerts directly. It has no dependency on the billing log. Tier two and three feed the same downstream alert pipeline. The separation is intentional. Real-time
[05:00] alerting should never have to wait for the billing data. So, tier one is deceptively simple and the place I would tell any team to start. Single REST API call. Get SQL history
[05:17] queries with status set to running. Token off straight from the notebook context. No cluster spin-up is needed. Near zero cost. Filter. Duration is in milliseconds, so filter anything past 15 minutes gets flagged. For each violation, email the
[05:34] user with a query text and duration. Audit row written to Delta for dashboards. Two practical notes. If you're running multiple workspaces, each one requires its own REST API call. Log all results into a common Delta table with workspace
[05:49] details. That's how you get a unified view across workspaces and avoid blind spots from any single workspace data. Second, outbound email requires a routable workspace. I route all alerts through the single workspace that has SMTP
[06:05] access. Don't try to send from each workspace independently. Start here before building anything else. This proves value in a single day. Ship it. Let stakeholders see named emails landing in their inboxes.
[06:26] Next, tier two. Reads from the billing usage table, the authoritative source for cost and DBUs with about 2 and 1/2 hours lag. Three things to get right. First, user resolution. Billing rows don't have one clean user column. We correlate across identity metadata. Run as, then created by,
[06:44] followed by owned by, then fall back to user ID. Order matters. Run as is one who triggered the work. Second, SKU classification. Billing usage has SKU name, a long descriptive string. Dozens of variants. We derive SKU type
[07:00] ourselves into three buckets, like patterns, inference, job, interactive. Three buckets, but different cost profiles, different thresholds, different urgency. Adding a new bucket later is one line of code, no schema change. Third,
[07:15] the aggregation trap. Individual billing rows log multiple small DBU entries per user in the system billing table. They will never cross any sensible threshold on their own. You must group by user, SKU type, and
[07:31] business day date first. Sum cost and DBUs and then apply thresholds. And anchor on your time zone, not UTC. Otherwise, a 5:00 p.m. spike lands on the next business day. One additional note,
[07:46] the cost join billing list prices is a built-in system table and works out of the box for most workspaces. If you are on an enterprise contract with negotiated rates, apply that on your cost logic.
[08:05] So, this is the core of the system. We are going to be solving the executor versus creator problem. By now, we know that billing usage attributes equal warehouse cost to the warehouse creator, the owner, not the person who ran the query. Tier three fixes that by working through three steps. Step one, the identity gap.
[08:23] Join query history to billing usage. Now, you can attribute cost to the executor now instead of the creator. This alone gives you 80% accuracy at 5% of the complexity. Ship this first. It's useful immediately. Step two,
[08:40] add warehouse events. Warehouse events tell you exactly when the warehouse was actually processing work. We attribute only during those utilized windows, which significantly improves accuracy. Step three, cross-hour attribution. Queries that run span hour boundaries
[08:58] with a proportional overlap calculation. Formula is on the screen. One important implementation note, once you have tier three running, exclude warehouse rows from tier two alert path entirely. Otherwise, you double count, once to the creator and once to the
[09:14] executor. Now, this is a bonus slide. So, before you turn a single threshold, check your warehouse size. This is the biggest cost lever in the system and the one most teams overlook. The
[09:29] formula is simple. Total billed DBUs minus what tier three attributes to actual queries. The delta is your idle time and over sizing overhead. If it is large, the warehouse is a problem, not your users. The math is unforgiving.
[09:46] Uh extra large runs at 13 times the cost of an extra small. Idle with zero queries running, it still builds at the rate. I had scaled our SQL warehouse up to extra large to handle a surge in usage. Reasonable call at the time. Within a
[10:02] couple of weeks weeks, it showed up on the bill in a way that was hard to ignore. By dropping it back to small, we saved more than 25% of the SQL warehouse spend. One change, no code, no pipeline work. Right size first.
[10:17] Tune thresholds second. Turning alerts on top of an oversized warehouse is the wrong order of operations. So, how this runs in production day-to-day?
[10:34] Three tiers to three workflows. Three tiers map to three workflows in production, plus one optional fourth that I'll come back to. Workflow one is the REST API monitor. It runs every 15 minutes during business hours at a very low cost per run. Anything running past 15 minutes gets
[10:50] flagged immediately. Workflow two handles job monitoring. Scheduled jobs are predictable, so rather than alerting on every run, this one sends one daily digest scoped to the business team. Clean signal, no noise.
[11:05] Workflow three covers interactive and SQL warehouse spend. It runs every two hours during business hours with two chained task. Task one runs the SQL warehouse attribution. Task two reads the output and fires alerts. Keeping them separate means you can
[11:22] recalibrate attribution without ever touching the alert logic. And the optional fourth workflow, it's a nightly reconciliation. This backfills any missing SQL warehouse rows via a merge on statement ID and warehouse ID.
[11:37] And this like if end of the day dashboard accuracy is a hard requirement for your team. These cadences are a starting point. Tune them to your team's rhythm. Now three design principles behind every decision.
[11:53] Modular. Attribution and alerting are independent notebooks. Add a new skew skew type, say model training, without touching the alert logic. Test attribution in isolation before it ever hits the alert path. Configurable. Every threshold is a
[12:09] widget parameter. Recalibrate quarterly or periodically without a code change. One notebook, multiple schedules, no hardcoded values. Audited. Every alert lands in spend alert log with email and status. Threshold crossed but dedupe suppressed.
[12:27] When finance asks, "Hey, why did cost spike last Tuesday?" You pull the log, answer it under 30 seconds with receipts. So dedupe matters more than most teams realize. Without it, users build inbox
[12:43] rules to suppress your alerts. And the behavior change is practically harder than most people realize. Dedupe key is technically user ID, skew name, violation type, alert date. For each of this combination, we show the cost at which we last sent an alert.
[13:00] When current spend reaches last alert plus threshold, we send again. One example, if threshold is set at $25, user crosses $25, alert fires. 50, another alert. 75, again. That's what drives midday correction, not waiting
[13:15] until the next morning. And one callout on service accounts. Always route to the admin distribution list. We don't filter them. Filtering hides automated overruns, which are exactly the ones you most need to catch.
[13:34] So, Cadence has to match the workload type. Interactive and SQL warehouses spend our budget. A heavy query at 9:00 a.m., you want an alert by afternoon, not the next morning. Workflow three, that's why it runs every 2 hours during business hours, so that like we can send
[13:50] that alert. Scheduled jobs are predictable. A daily digest is enough. No overrun noise for work you already know is going to run. One distinction worth internalizing, schedule and scope are not the same thing. Workflow three runs every 2 hours, but
[14:06] always looks at the full business day. Each run builds on the previous, not a fresh 2-hour slice. Get that right, and you your alerting stays coherent, regardless of how often you pull.
[14:24] Alerts tell you why. Alerts tell you when a threshold crossed. Dashboard tells you why and where to look next. So, we can start with four dashboards, all built on the same Delta tables powering the alerts. Zero new ETL shipped in one sprint. So, the first one is cloud spend
[14:39] overview. DBU and dollar trends by skew type, week over week trend. Next, SQL warehouse usage, broken down by AI/BI dashboards, catalog, notebooks, apps, and like rest of the stuff. Next, alert trends. What is the violation magnitude over
[14:56] time? This is how you prove the system is changing behavior. Next, user spend leaderboard. Actionable targets for coaching conversations. One finding that discovered us, after backfilling 3 months of SQL warehouse data, the top cost driver was AI/BI
[15:13] dashboards running full table scans with no date filters. That shaped how we think about dashboard authoring across the whole org. The system built context, not just alerts.
[15:29] So, this is what happened when we turned this on. 51% reduction in the SQL Warehouse built skew cost within 30 days. Driven primarily by right-sizing and attribution emails landing the right inboxes. 14% reduction in the serverless spend sustained over 90 days. 57%
[15:47] reduction in violations from month one to month three. The 57% is the one that I'm proudest of. The first two are cost numbers. The 57% is the behavior. Users changed how they work because they received a named costed email with a
[16:03] specific query in it. Your query ran for 47 minutes and costed $87. Team-level aggregates don't do that. Named attribution does. And the finance Slack messages, "Why is the cost spiking again?" Stopped.
[16:23] So, are we ready for four things for your Monday morning? Number one, check your SQL Warehouse size before tuning any threshold. It's the fastest, highest cost leverage in the system. Two, tier one REST API alerting is already available for you right now. Near real-time, no cluster cost. Start
[16:40] there. Third, attribution beats aggregation. So, specific email about your query costed $87 and ran to over change the behavior. A team-level spend doesn't. Four, build a dashboard. See what's being sent, who
[16:55] is violating, whether behavior is actually shifting. Visibility into the monitoring system is what sustains it. Whether you manage hundreds, like thousands, or millions in Azure Synapse Analytics spend, these four patterns move you from reactive to proactive.
[17:12] That's a shift. I would like to extend special thanks to Usha Kiran who's here on AEP team, Nitin Sanghi on CFM Ops, Dhirendra Patel and Kushal Bora on Deep as team for the recommendations support while building
[17:27] the system because this was a side project for me. I'm very grateful for the collaboration. Let's connect and you can find me in LinkedIn. Happy to compare notes if your team is building something similar. Thank you again for joining me on this
[17:43] talk today.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.