From BigQuery to Databricks SQL: 50M queries, 40% cost savings, 7x faster

Summary

Nubank migrated 50 million monthly SQL queries from BigQuery to Databricks SQL over 12 months across 17,000 analytical users, achieving 40% cost savings while maintaining zero downtime during a period of explosive growth.
Performance improvements included 7x faster P90 query times enabled by serverless compute, liquid clustering, and workload isolation, while Unity Catalog maintained governance and audit trails required by financial services regulations across multiple countries.
AI-assisted query conversion tooling—starting with SQL Glot and evolving to Genie Code—accelerated migration by handling dialect translation, enabled by a prior year's Unity Catalog migration that decoupled data ingestion from workload migration.

From BigQuery to Databricks SQL: 50M queries, 40% cost savings, 7x faster

Migrating large-scale SQL workloads to a new platform requires more than engineering effort. At Nubank, managing 50 million queries per month across 17,000 analytical users in a highly regulated environment demanded a strategy that balanced cost reduction with zero downtime during explosive growth.
This talk explores how Nubank successfully migrated from BigQuery to Databricks SQL over 12 months, achieving 40% cost savings and 7x P90 query performance improvements. Learn how they leveraged serverless compute, liquid clustering for data organization, workload isolation, and AI-assisted query conversion to scale to 50 million queries monthly while maintaining governance with Unity Catalog in a highly regulated financial services environment.
🤝

Chapters

FAQs

Why did Nubank migrate from BigQuery to Databricks SQL?

Nubank migrated to consolidate SQL as a first-class citizen alongside the Databricks notebooks and Spark workloads their analysts were already using, while also targeting cost reductions and improved scalability. Their analytical users were already familiar with the Databricks Data and AI platform, making SQL consolidation a natural next step.

How did Nubank handle query migration at scale?

Nubank used SQL Glot, an open-source query translation tool, to convert BigQuery SQL dialects to Databricks SQL, and later adopted Genie Code for AI-assisted migration as the tooling matured. Migrations were executed in waves with dedicated support infrastructure and stakeholder communication to avoid disrupting analysts' workflows.

What performance improvements did Nubank achieve after migrating to Databricks SQL?

Nubank achieved 7x P90 query performance improvements driven by serverless compute, liquid clustering for data organization, and workload isolation that provides resource guarantees. Liquid clustering reduced query times from hours to seconds for key workloads.

How does Unity Catalog support regulatory compliance at Nubank?

Unity Catalog provides audit trails for queries and permission changes that satisfy the regulatory requirements of multiple countries where Nubank operates. The governance foundation was established in a prior migration year, allowing the SQL migration to focus entirely on workload consumption rather than data ingestion or access control setup.

Full transcript

[00:08] Hey folks, first of all, thank you everybody for being here today. Uh, this is my first time at the AI Summit and I'm really glad to have all you folks here with me. And today we are going to talk a little bit on how Nubank uh,
[00:23] performed a huge migration from warehousing from BigQuery to Databricks SQL while increasing performance and reducing costs. Uh, first of all, I'm going to talk a little bit about
[00:38] Nubank. Nubank is one of the largest financial service institutions in Latin America. Currently, we have more than 130 million customers uh, across multiple geographies. And we have as our mission to fight
[00:56] complexity into empower to empower people. And and our scale it means to have uh, scalable uh, reliable and governed data. Uh, we wanted to migrate of course
[01:11] because we wanted to have cost reductions. But not only that, we wanted to have a better future on scalability. And we also wanted uh, to provide a more unified user experience to our users as our
[01:29] analytical users were already used it uh, with Databricks. But on using notebooks with Scala and Spark code, not actually SQL. So we wanted also to consolidate SQL as a first-class citizen
[01:45] within Nubank for analytics. And the SQL warehouses with Databricks SQL helped us with that. And we had a key enabler for our We were able to only focus on actual
[02:01] workload migration because we were privileged to have our colleagues in the prior year to our migration to migrate all of of your data our data in Databricks to use Unity Catalog. So, we only needed to work on
[02:18] the consumption of workloads itself, not data ingestion. And we had some challenge to overcome. First of all, we are a company at a growing scale.
[02:33] By the start of our migration in 2025, we had about 20 million monthly queries in BigQuery alone. And across our migration, this number
[02:49] got much, much higher, actually. And our users, they work on different scopes. We have users that work with ad hoc queries, with one-off dashboards,
[03:04] with KPI tracking, with regulatory solutions needed to work. And uh we could not just let disruptions happen to their workflow cuz this would cost a lot for them.
[03:22] And what it meant was that we couldn't ask our users to be slow down during migration. And also, as we are a financial service institution, we are highly regulated. So, we must
[03:39] conciliate the regulations of multiple countries, making sure that all of them are being complied while working with analytics. We have We must have audit trails for our queries, for permission changes, for everything that that happens within our
[03:56] environment. And almost half of our users were reporting bad performance on the legacy platform. So, it's not just a matter of us wanting to migrate, but also users wanting
[04:13] a better solution for their day-to-day work. And how did we get there? Well, first of all, this was our dream feature. Uh and actually this was the future that we actually were were able
[04:28] to to get. Uh in this future, we have Unity Catalog as the central governance on top of everything that we do. We have multiple workspaces. Uh for each Databricks workspace, we have a
[04:44] a set of warehouses. In our case, uh these warehouses are split up by business units. So, each business unit has a isolated compute uh resource to work with. And all this set of
[05:01] uh warehouses power our consumption layer for our GRCs, for apps, uh reports, and BI dashboards, operations, and external APIs such as MCPs, for instance.
[05:25] And to achieve that, we actually we broke this migration in five phases. Going from enablement to actual sunset. On enablement, we wanted to make sure that all of our users were able to use Databricks SQL, no
[05:41] matter who they are. The first thing that we decided to do was to make sure that we had a parallel operation between BigQuery and Databricks SQL so that uh the users could start migrating whenever
[05:58] they were capable to do so and not have a big bang change. This allowed us to have a manageable throughput uh that we could adjust to our user needs
[06:13] and to make sure that we would provide a good enough uh level support to them during the migration. Uh One thing that was really important to us was to provide day one access to Databricks SQL to all of users.
[06:30] So, even if uh for cases that didn't need to be migrated uh already the users would be able to use Databricks SQL. That meant that uh they could start leveraging ad hoc
[06:45] analysis with it which in our case meant users leaving notebooks with the need to write code on them to actually just write queries.
[07:03] And with that, we were able to continuously learn about our users, about our platform, understand uh how resources uh worked for us, how we would fine-tune them and actually be able to at each
[07:19] iteration have better and better a better and better platform for our team. And also, we provided proper tooling for the users that would like both to experiment and to migrate stuff. Uh first of all, uh we provided an
[07:36] assistant page for the users using a lib called SQL glot. For those who don't know this leap, it is uh a tool that allows you to convert SQL dialects, and it does support both
[07:51] BigQuery and Databricks SQL. Basically, it creates a parsing tree of the the queries, and then you can switch around to the dialect you want. And this is cheap. This is deterministic. And it really was really good for us to
[08:09] use. Uh users also had over the course of the migration access to AI agents to help, uh specially with fine-tuning and uh error solution.
[08:24] And also, we made sure that our users had access to migration tracking of all dashboards, all assets that they needed uh to to move from one platform to another.
[08:41] We also wanted to optim- optimize our time. We didn't want us to get everything that was in BigQuery and move to Databricks SQL. Cuz there there was a lot of stuff already in BigQuery that was not used anymore. We had it up and running for more than 5 years in Nubank. So,
[08:58] as teams grew, teams ceased existing, a lot of things uh were created and died without being cleaned up there. So, the first thing that we did was a cleanup on the legacy platform. We
[09:15] created a an inventory of all assets that depended on SQL workloads. We managed to understand which were the owners, which were the creators of each of these assets. And we used some heuristics to define
[09:34] which workloads were actually deprecated. So, we can we could remove them from the ask to our users both to reduce the effort that they would have to make it, but also to reduce our effort in supporting this
[09:51] migration. Uh as I said, uh we allowed users to use SQL glot for acquiring migration. And they were not actually using just the tool, but actually we
[10:06] uh constructed a Databricks app at the time uh that had an interface with SQL glot where the user could just copy the the previous query that they had, paste there, and receive a new a new query.
[10:24] And uh we also tried to optimize a bit of these queries. Uh with SQL glot, we were able to uh have a very clear view of what what were the joins that were happening in
[10:39] the in the queries, uh the orders of filtering, the cities that were being used. And we could uh define patterns to have queries that if were not at the same level of the previous
[10:56] queries were actually better both in terms of performance, but also in terms of a standardization. Then, uh we needed to do to establish a pattern on how we would migrate everyone.
[11:11] Uh first thing that we knew from day one was that we would need to have access control. And we were lucky because data uh Databricks with Unity Catalog provides that for us.
[11:26] And having uh our SQL warehouses splitted by business unit allowed us to isolate their workloads and actually enable future cost allocations as well. We defined our waves of migration and we
[11:44] broke them down by use case trying to group together cases that were similar and teams that worked together so that when the migration ended the user were actually
[12:02] able to stop using the legacy platform. And also we leverage Terraform to pre-provision all the warehouse that we needed and by the start of the
[12:18] migration we already started with more than 400 SQL warehouse up and running. And none of that actually provided us any upfront cost because as we are using serverless with auto termination
[12:36] until the users actually started using these warehouses we are not being billed by it. Then we started to actual actually migrate the waves. And we had a a very structured life
[12:54] cycle for for these waves in which one or two months prior to the start of the migration we would notify all the users and specially the stakeholders of the teams that would
[13:11] need to migrate their workloads so that they they could estimate the efforts and they could actually allocate head count to to change everything everything that was needed.
[13:29] We also provided direct lines of support for uh our customers and this this was in the form of Slack channels, office hours, documentations, guided trainings so that we could as much as we could
[13:45] provide a smooth path for for them. And although we uh I believe we created a very good plan on migration there will always be some long tail to be migrated. Always some
[14:02] minor case that will exist and that will not be able to be addressed on the time period that is defined. And to address that, we made sure that we had direct lines of communication with the stakeholders responsible for
[14:18] them so that we could close the gaps as soon as possible and don't get to the end of the migration with a snowball. And by the end, we actually needed to turn off the Lexer platform.
[14:34] For us, it actually was one of the easiest parts cuz as we did this wave migration with users changing their workloads and starting to work solely on Databricks SQL
[14:49] actually each each step was reducing a little bit the amount of workload that we had in BigQuery. And when we get to the last wave we had basically nothing there. Uh so on our last mile of migration we
[15:08] kind of basically pulled the plug and we didn't hear anything. Uh we didn't have any surprise. We didn't need to do any kind of rollback.
[15:25] Well, first of all, I said at the beginning of the session that we had uh some performance improvements. But, why? First of all, serverless compute. Uh having serverless compute uh with their warehouses helped us a
[15:41] lot. It's kind of counterintuitive because we think, "Oh, the the scale scale for uh serverless is more expensive than using Pro, for instance." But, actually, the amount of uh
[15:59] time that we get back from having very fast uh scales up and down and auto terminations with uh local starts actually uh offset it for us uh
[16:14] the cost the this extra cost for serverless. Uh for you folks to have an idea, currently, our auto terminate for our warehouses is of 5 minutes.
[16:30] And any warehouse spins up back again in like 4 or 5 seconds. And another really important thing is liquid clustering. Liquid clustering provides uh very good uh performance for partition
[16:46] data. Uh and for uh workloads that had liquid clustering uh enabled, we have seen queries going from hours to seconds to run. And when I said it here, hours, it was
[17:04] really hours. It's very common at Nubank that we have uh source tables with two, three, four, five terabytes of data. And also, and maybe the more the most important here for us was workload
[17:21] isolation. Uh in BigQuery, we had uh just a few slot reservations uh shared by multiple teams. Uh with this isolation in Databricks,
[17:36] we allowed the the teams that were smaller to not be actually dwarfed by by the teams that had multiple multiple queries running. But not only that, for the teams that
[17:52] are big, they can rest assured that when they request uh more resources for their workload, these resources go directly to them and not to cover some other team.
[18:10] Governors from day one is a lifesaver. Unity Catalog provides us with usage telemetry at every level that we need. And combined with system tables, uh we can govern an access and have cost attribution at the the granular state we desire.
[18:33] Also, data layout is key. Uh with uh BigQuery and GCP, uh we had a an issue that was our data is in is in AWS. So, all the data for analytics needed to be
[18:48] moved between clouds. And not only it's slow, it's costly, but also, it has a side effect that when we loaded this data to BigQuery, it actually became indexed
[19:03] tables. So, it is somewhat resilient. Uh with Databricks, uh we directly ingest the data from S3. So, what it means is that uh,
[19:19] we don't have networking costs for uh, cloud transfer. We don't have this time to load the tables. But, we are reading from an object store. If we don't uh, don't be cautious uh, about the data that we are writing
[19:35] there, we we can incur the risk of doing full full table scans all the time. Uh, that means that writing data uh, already with good clustering good in good partitioning uh, is a world difference
[19:53] on the performance that you may have. But, when you do that, the performance is really better. In practice, when you do have this uh, right uh, mindset with the partition and
[20:09] clustering, you can have queries going from hours to seconds. You have liquid liquid clustering running all day long uh, and continuously optimizing your queries. And and
[20:24] if you have the that data written in uh, with partitioning in mind from day one, you have a much easier time. Also, uh, we talked a little bit about uh, the assistance that we had. But, we started
[20:42] our migration uh, in early 2025. The world there was completely different. Yeah, there was uh, AI assistants. There was there were enterprise level AI
[20:59] assistants, but most companies didn't have access to them. When they had, most of their users didn't how to use them. And the engineers didn't trust their results. And for sure I was one of them.
[21:21] But we currently have a completely different world today. In early 2025, we are focusing on SQL glot because it was deterministic. It was rule-based. And we just could put it uh on uh behind the web page.
[21:38] And it was really good actually. It handled about 80% of our queries that needed to be migrated. But by mid-2025 uh 2025, we already had uh more context on how to use
[21:56] uh AI tools. And we have seen by that time users starting to use Genie code, which actually at the time was called Genie Assistant, to
[22:14] get the query that was migrated for them, put in in the SQL editor, run, maybe get an error, and ask the assistant to help them finding what which was the way the error and fix the query for them. And this actually helped uh to cover the other 20% of queries that
[22:31] had issues uh to to be converted. But in 2026, the world's different. Many many companies have access to LLM agents. They have access to skills to NCPs and all that.
[22:48] And today uh we can have end-to-end orchestrations for all of that. And so I would like to show you an example of what could be done today. And here we have a a very simple example
[23:07] of a framework on how we could leverage AI agents to have query migration. We could have an LLM agent such as Cloud, Cursor, Genie Code uh
[23:23] managing uh SQL Glot. I would never list SQL Glot. It's cheap, it's fast, and it works. But we can connect it to the agent. Uh we can have path adapters with uh skills
[23:39] uh so that you can input your business context on table names on uh some metadata, some uh guard rails that you want to have on the on the new queries that you are converting.
[23:55] You can connect your agent to Databricks MCP and ask it to actually run the the new query and check if he if it actually runs, the results, how much time it takes to run. Uh you can get your legacy platform
[24:13] connected there as well and and get uh the benchmarks on how they run. Uh you can get Unity Catalog metadata on top of that. And then you can package all of that
[24:29] into for instance a single skill, a Databricks app, whatever and give it to your user and they will just input the query, will not have to do any manual work, and get a query that is validated, that is
[24:44] working, that is benchmarked on the other side. Uh which was the pay the payoff actually of our migration? Well, first of all uh we had more than 5% of savings on
[24:59] compute when compared to Thank you. When compared to BigQuery and there's a little caveat here that actually is a good caveat that
[25:15] it only considers compute to this number. We are not talking about the amount of data that we stopped transferring between clouds. So, actually this the savings are actually way way huger.
[25:33] And it's not just the cost but prior to that we had shared lot reservations. We need to have heuristics to allocate costs. We didn't have granular attribute attribution. We couldn't for instance
[25:49] provide analytics on top of private personal data because we don't didn't have the governance there. But with Databricks SQL and Unity Catalog we can provide all that. We have execution time telemetry of everything
[26:04] that happens. We have a precise chargeback and we don't need to create crazy infrastructure to do so. Also, we've seen a 7x P90 improvement in our queries getting to a P90 of less than 3 seconds.
[26:23] And this is a It's already a good number. But there's also the caveat that most of our data doesn't run on liquid clustering. So this 7x time is with a lot of full table
[26:41] scans. So, I would I wouldn't find it crazy if we said that if everything had liquid clustering it actually would be like 70 times.
[26:57] And these are some numbers of our platform currently. We have more than 400 SQL warehouses running. We have a very old number here of data apps. Here we are saying that we have more than 170 data apps running, but actually as of
[27:14] last week, we have more than 400. And going up. We have more than 800 active Genie spaces. We had this more than 40% cost savings.
[27:29] Currently, we have more than 8,000 dashboards powered by Databricks SQL. We have more than 50 million queries per month running, which is a 2.5x increase in 1 year.
[27:45] We have this sub 3 seconds performance. And we stopped loading more than 50 petabytes of data monthly between clouds. And considering all these numbers, all all I can say is that we're just not
[28:02] we're not just surviving with this platform, we're actually thriving with it. Woo! Thank you. Yeah, that was basically it. Uh if you if it resonates with you, if you
[28:17] have any questions, please feel free. We still have some time if you have some questions now. Please, otherwise you can find me later and we can chat. Let's go, Brazil! Woo!

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.