Improving User Experience and Efficiency Using DBSQL

Summary

• Serverless scaling: PicPay scaled Databricks SQL to 2,000 users by adopting serverless compute, which automatically scales resources during peak demand and reduces waste.
• Governance model: The bank implemented continuous monitoring for query and dashboard performance, deactivating access to queries that exceed time or impact limits.
• Efficiency gains: Tight content governance combined with serverless scaling brought greater efficiency, cost reduction, and an improved user experience for PicPay's workforce.

Improving User Experience and Efficiency Using DBSQL

Talk by Gustavo Tadao Okida, Head of Data Platform and Governance, PicPay ; Renato Suarez, Data Platform Manager, PicPay
To scale Databricks SQL to 2,000 users efficiently and cost-effectively, we adopted serverless, ensuring dynamic scalability and resource optimization. During peak times, resources scale up automatically; during low demand, they scale down, preventing waste. Additionally, we implemented a strong content governance model. We created continuous monitoring to assess query and dashboard performance, notifying users about adjustments and ensuring only relevant content remains active. If a query exceeds time or impact limits, access is reviewed and, if necessary, deactivated. This approach brought greater efficiency, cost reduction and an improved user experience, keeping the platform well-organized and high-performing.
Here's more to explore:
The best data warehouse is a lakehouse: https://www.databricks.com/product/databricks-sql
How to migrate legacy data warehouses: https://www.databricks.com/resources/ebook/migrate-your-legacy-data-warehouse-databricks
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How did serverless compute help PicPay handle more Databricks SQL users?

Serverless automatically scales resources up during peak times and scales down during low demand, preventing waste and enabling efficient resource utilization. This solved the problem of DBSQL Classic not being able to handle the volume.

What is PicPay's governance model for query and dashboard management?

PicPay implemented continuous monitoring to track query and dashboard performance. If a query exceeds time or impact limits, the team reviews access and deactivates it if necessary, keeping content relevant and the platform high-performing.

What business results did PicPay achieve with this approach?

The approach delivered greater efficiency, cost reduction, and an improved user experience while keeping the platform well-organized and high-performing for 2,300 Databricks users.

Why is content governance important for a large user base?

With many users and queries, ensuring only relevant content remains active prevents platform bloat, maintains query performance, and helps users find and trust the dashboards they need.

Full transcript

So hello, thank you for being here. Um, who we are? We are from Pikpay. Pikpay is a digital bank in Brazil. Uh, one of the largest one in there. And our object objective is to simplify our customer financial life providing products in the right time in the right pricing in a the better way to them. We use data and technology for that and we are obsessed to provide the best banking experience. uh some important point to us, we are pioneer in implementing instant payment back in 2012 12.
So it was a peer-to-peer payment. Um, today we work with the Ulta personaliz personalized experience providing more than one 1,000 1 million different homes according to the customers behaviors, customer data that we have and we are referencing Brazil in using open finance and generative AI. uh to give you some context about how we are to the market. Uh Pix is an instant payment in Brazil, right? And um we are proud of being of having more than 11% of the PIS transactions has at least in one of the edge, you know, one of the end a pig pay account.
So it's a it's massive amount of pigs. We have six million of consumers using click pay. uh almost $50 billion in cash and in the last year we transformed ourself in a financial AI company for thinking AI first and at the end we we are very proud of being chosen by a popular vote uh to be the best digital bank in Brazil. uh some challenge internally we work with different business units in a very autonomous way. So they have a lot of autonomy to build products to improve products and generate data.
U so in the other hand our challenge is to keep controlled right to keep controlled. Um during the year we during the last years we acquired some companies and we we merged with other comp companies and then we have uh we so uh we we brought a lot of different data tools inside and one of our challenge is to keep the governance and then we uh almost centralize everything in data bricks. And uh about the who who are using internally we are about 4,000 employees. uh more than 50% use a data tool and 2300 users are data brick users in big at pay that represents oh almost 6% of uh of our employees thank for this opportunity today I'm will present our data platform case well you probably noticed pay has been used data bricks more and more and that brought some consequence.
We reach a point where the datab bricks SQL classics could no longer handle the volume of the queries and this cause two major impacts. First our users start complain that queries are taking longer to run and this impact the data analysis routines the report updated and we saw a 30% increase related to the cluster perform to deal with the situation. Our teams added more resource to the clusters and this impact our platform cost and we saw a eight 8% rise in the average daily costs.
It was a challenge situation. So what did you do about it? Well uh we start the implementation of data brick SQL serverless and we quickly realized that with a solid architecture and a strong data context governance we had a great opportunity to change the game and exactly what we did the results were very positive. Uh now our users were running queries much faster. The performance is up six time better compared to the class setup and also we saw a reduction of up 44% in the average daily costs.
Now not made our users happy but give us a real financial efficiency. Uh let me show you how we got there. To achieve it these positive results we took four key steps. First we adopt the datab bricks SQL serverless. Second we rethink our class architecture. Third we implemented the automatic report load balance systems. And finally we created a strong data context governance policy. In in the next slides, let me show each steps of this process.
In in the old tetra, we had a single datab brick SQL classics clusters. It was large, expensive, inefficient, handle all types of the quaring dashboards in the single clusters. It leads to high competition for resources. And now in the new setup we have a five smaller datab brick SQL serless clusters that's working together to handle the vellum corus more more efficiency but to make this architecture work we need a way to manage how the queries were roted across the clusters and we create a new process to handle that.
Let me show you how it works on the next slide. Oh well uh we implemented the automatic report load balance systems when users run a query or refresh uh dashboards. The process capture the working loads and distributes across the available clusters. Now we are able to process a high numbers of the queries in the same times without overloaded environments. Uh however we need a way to separate the low performance squares and dashboards from the optimized ones.
Let me show you on the next slides how it works. uh we implemented the data governance policy that monitor our environment continuously. If a dashboard or query run with the performance limits, we tag as approve according the pol the performance policy but it's exceeds the performance limits. We tag as not approve notify the owner on the slack and move the content to another cluster and keep the environment efficiency.
These four key steps together help us change our architecture process and future and made our environments grow more efficient and susable way. Okay. So as next journey so we we somehow could handle the problem of using SQL at scale right more than two 2300 employees at uh at big pay are quering the data lake so it's something that is huge and as next steps we are implementing the no code reports So using Genie. So we we we discovered that some specific group of users that use a lot but with very um the questions are are very similar.
We are able to provide a very good solution using Genie. So it's it's the results are good. uh to another group of users we are introducing the AI functions in the SQL so they can take the advantage of uh no using AI uh in in the query that's fine and at the end uh for the other groups we are implementing the mood agent to create guard rails uh to to answer correctly to generally So this is what we are building and um for those that those new employees that are coming or for to to for the others we are thinking in a program that's called data champions data bricks evolution is to train the the population there is to give more empowerment in building better solution.
Uh the learning process is based on datab bricks um content that's provided by datab bricks and we create several comm community communities internal uh to share and to study to share and to learn and in the other hand we are very proud of being part of databicks data product a board advisor where we share with them our challenge we share talk to with them uh how we how we suffer right with uh some case there and then we are able to discuss what's no how how it's going to be the next feature right so thank you very much and if you have any question we are happy to answer thanks so much thank you

Learn more about the Databricks Data and AI platform.