Rebuilding Brazil's Payment Foundation: Cielo's Databricks Transformation

Summary

Cielo, a Brazilian payment processor handling 8% of Brazil's GDP through 700,000 merchants, migrated from a fragmented environment of AWS EMR, Oracle Exadata, and SAS—with 7,000 daily jobs and 900 undocumented PL/SQL programs—to a unified Databricks lakehouse over two years, reducing ingestion pipeline costs by 40%.
The migration used a coexistence strategy that maintained legacy systems during the transition, and a cultural transformation program called Brilliant Futures shifted the organization from siloed projects to AI-as-capability embedded in daily workflows for 1,000+ self-service users.
Three integrated capabilities were deployed on the platform: analytics for democratized insights, conversational intelligence through Genie for natural language data access, and governed AI agents that negotiate directly with customers via WhatsApp.

Rebuilding Brazil's Payment Foundation: Cielo's Databricks Transformation

Cielo, processing 8% of Brazil's GDP through 700,000 merchants, faced a foundation crisis: fragmented across AWS EMR, Oracle Exadata, and SAS, with 7,000 daily jobs, 900 undocumented PL/SQL programs, and 100+ teams working in silos. Trust was broken, different tools producing different numbers for the same question. Over two years, Cielo migrated to a unified Databricks Lakehouse, enabling three integrated capabilities: analytics for democratized insights, conversational intelligence through Genie for natural language access, and governed agents for autonomous decisions.
this video covers the migration strategy balancing coexistence (maintaining legacy systems while building new) with enterprise adoption, teaching 700,000+ users a new language for data. Key outcomes included 40% cost reduction in ingestion pipelines, 1,000+ enabled self-service users, and deployment of conversational agents negotiating with customers via WhatsApp. Cielo's Brilliant Futures program transformed the culture from siloed projects to AI-as-capability embedded in daily workflows. The governance-by-design approach through Unity Catalog and the data-framework-first philosophy proved essential: AI only scales when the foundation is trustworthy and the organization is ready.
🤝

Chapters

FAQs

What was the data foundation problem Cielo needed to solve?

Cielo's data infrastructure was fragmented across AWS EMR, Oracle Exadata, and SAS with 7,000 daily jobs, 900 undocumented PL/SQL programs, and 100+ teams working in silos. Different tools produced different answers to the same question, breaking trust in data across the organization and blocking the company's ambition to scale AI capabilities.

How did Cielo migrate to Databricks without disrupting payment operations?

Cielo used a coexistence strategy that maintained legacy systems running business-critical payment processes while incrementally building the new Databricks lakehouse. Given that Cielo processes around 8% of Brazil's GDP, any disruption to operations would have significant economic consequences.

How does Cielo use AI agents in production?

Cielo deployed governed AI agents capable of autonomous customer interactions, including conversational negotiation with customers via WhatsApp. The agent capability layer is built on the Databricks Data and AI platform with governance-by-design to ensure that autonomous decisions remain within defined business and compliance boundaries.

What is the Brilliant Futures program at Cielo?

Brilliant Futures is Cielo's internal cultural transformation program that shifted the organization from working in siloed data projects to treating AI as an embedded organizational capability. The program supported the enablement of more than 1,000 self-service analytics users and the adoption of Genie for natural language data access across the company.

Full transcript

[00:07] Hi everyone. Uh I'm glad to be here to talking about our journey. So, my name is Gabriel and I will tell you about our journey to the data and AI unified platform. A data and AI unified platform to scale
[00:24] AI capabilities for all the company, for Cielo. So, first about me. I'm leading the data and AI transformation at Cielo and my seat rolls between the legacy modernization and how can we
[00:41] enable AI capabilities for the companies, autonomous agents, conversational intelligence and scale advanced analytics and machine learning. So, uh for those who don't know Cielo, Cielo is
[00:57] a payment company in Brazil. It's a benchmark in Latin America and we are um in almost 100% of Brazilian territory. So, we have more than uh seven
[01:13] 700,000 customers from different segments. So, we work with digital e-commerce, large enterprise, medium enterprise, small business. So, all the retail segment are working with Cielo.
[01:29] And we are processing around 8% of Brazil GDPs in our company. So, I'm telling you about these numbers because when we think to change something inside the company,
[01:45] we need to think in all the ecosystem. Uh how we can change without impacting our customers or impacting some point of economy. So, we had ambition. We need to rethink
[02:01] our environment to enable AI capabilities for all the the company to enable a better experience for our customers. And when we look at to our environment to rethink our capabilities, how to scale AI with
[02:18] trust, we look at that data wasn't the problem. We had a lot of data for all the data transactions with for all the customers. So, what's not the problem? AI wasn't the problem, but we have a lot of
[02:34] models. The models are improving up day by day. And what's the problem? The foundation that we are running to data and in AI. The foundation was the problem. So, we need to rethink this. We need to rethink
[02:49] how can we scale to all the company this foundation. And our ambition was moving faster than our architecture. So, when we start the reality was we have a complex environment with
[03:06] business process running every day, every night, but we didn't have a confidence uh after the business process. Uh the scale the scale we couldn't scale because we have a lot of silos of data.
[03:22] We'll show you the scenario that we are we are working we were we're working. And our ambition was scale AI capabilities for all the companies, makes autonomous decisions, faster decisions than we did.
[03:37] So, okay. Uh we have our ambition. We we know that we have a a foundation problem. So, let's take a big picture. What was the scenario that we we encountered? That was the scenario, the initial
[03:52] scenario. Different kinds of technologies, different kinds of technologies to make decisions using data, using advanced analytics, using AI. Different vendors. So, we had a problem with data governance
[04:08] here. We had a problem to a lot of silos of data, and how can we scale AI using this kind of environment? Here was complexity. Uh we are not operating technology tech-
[04:24] technology here. We are operating complexity here. So, we have to change. Uh okay, let's go underneath in in in this scenario. So, how big is this is this scenario? We We are running in that environment
[04:42] around 1 petabyte of data, more than uh 7,000 process jobs running daily to make decisions, and around uh 900 PL/SQL programs with false documentations to generate
[04:58] business decisions, to generate business process for all that that uh with all the transactions that I've told you for uh about Cielo. So, and we we didn't have documentation here
[05:14] when the that environment was working around 20 25 years. So, we have just memory, people memory to to know, to rebuild the process. And the most problem, more than 100 teams are were working in
[05:32] that environment. So, was not just a techni- a technical problem, a technical challenge. Was a cultural change, too. We need to change the culture of the company to think with data with AI using data that we can
[05:48] trust. And trust trust was the point. Without trust, we can scale our our decisions. We can scale AI. We can scale autonomous agents because if you don't trust in our
[06:04] foundation, we don't trust in our data, we can scale the process. So, what happened? Well, different things with different numbers for different purposes. The numbers doesn't match when we need to to to make
[06:19] some decision. And that was the problem. How we can scale AI if just a report doesn't match with a a simple information. So, we are we're losing the battle with the governance. We We couldn't get
[06:36] governance with in that environment. So, we know the we knew the the the how big was the problem, and we know the way that we want to follow. Our
[06:51] ambition was saying to rethink that foundation. So, then we start to uh to talk with some vendors, to talk with some opportunities, and we need a foundation that we can governance the data, governance AI, uh
[07:08] assist the foundation, uh governance system is standard for all the companies, for all the process. We need to scale trust, scale of availability for data, for AI, and turn AI as a capability, not as innovation.
[07:24] So, we stopped to treating AI as innovation and think to treat AI as a as a capability. Then we start to talk with Databricks. Start to talk with Databricks because Databricks could help us to migrate all
[07:41] those technologies and the platform was delivering all those three pillars here. Then we start a migration 2 years ago we start a migration from Exadata SAS
[07:59] AWS Cloudera a lot of a different technologies and vendors here. And the first thing we build a foundation and set the governance standards that we will we work in.
[08:14] So we need to think and any red operation to work with data to make decisions. And we we need to think how can control and coexistence both the two different environments. The
[08:31] new one with Databricks that we started to build and the old one. The old one with complexity and we have to make sure the decisions were we're not impacting the business. So this control
[08:47] coexistence was very important to don't make any or impact for our customers or for the all the Brazil society and uh And the the another challenge was the enterprise adoption.
[09:04] We need to talk with people to rethink the way that they were working with data. Rethink the way that were make decisions with data and to be a data first and AI first company.
[09:20] So we need to work with them to teach them a new language a new language to to talk with data to to build their their models their programs and how can how can how they can use that the new
[09:37] platform. So, when we when when we start to build this and to transform the business operations, we start to generate value, too. So, we we did this along the two last years to
[09:54] start to generate the value here. And when when we move for for a platform that we can trust in data, we can trust in our decisions, we enable another different
[10:09] different capabilities for the company. This year, we work with three different capabilities. We we need to enable CL analytics, CL conversational, and CL agent. We need to enable analytics, conversational, and and agent, not like
[10:27] a project, like a capability for the business operations and into the workflow. So, trust transform us AI, not an experimentation. Transform us AI to using the business process into the workflows, and
[10:44] here was not a different capabilities running in the same foundation. So, I'll tell I'll tell you about something about the three capabilities here. And uh first of all, CL analytics.
[11:01] Here is not rocket science. Everybody do doesn't do analytics from years. But here, the different was confident, was trust, the way that we trust in data. So, the old way was CL
[11:17] was just looking at the the mirror uh rearview mirror and say what happens in the past, but we we could we couldn't influence the future. So, we start to rethink, rebuild this to influence the future and come up enable the teams to
[11:35] work with those models. We democratize the the build the build and the the development of models to influence right from the business. So, the CLOE was to scale capabilities of data and analytics here.
[11:51] And um we started to work with them and to scale the knowledge, to scale the platform and the behavior of the models, of the the decisions made of models
[12:06] changed to we enable to deploy the results of model right for our system, for our products because we are confident about the the results, the the data, too. So, we scale the capability for all the
[12:23] business process of the problems with data. This is a a change that we could um transform the new foundation, too.
[12:39] So, the next cup conversational is democratizing the access to data. The new language to work with data. So, the users are able to talk and question in natural language to uh to receive
[12:55] response from the from their business needs right from the data. So, they we here we offer the Denean Databricks one for for the for the sales team, for financial team just to ask questions.
[13:11] But, it's not a magic trick here. We have to build pipelines. We have to build context with data to show the teams what they need to be uh precisely respon- response to. So, here the true inclusion. Here democrat
[13:27] we democratize data for all the company and they could access the data without technical knowledge knowledge meant. So, the next step is
[13:43] if the our employees our all the people that are using data to make decisions here, why we can offer for for for our customers. So, we enable to show our customers the
[14:00] single source of truth. The same knowledge made base that we are using for conversational with conversational with our data, the the customers can receive the same information. So, here is the trust in
[14:17] the in the edge. So, when when a customer from Cielo access the the site or access access the app and make questions about their business, have the same response than the employee when asks about their the their mission
[14:34] merchants. So, the conversational here was was for the our customer and our employees, too.
[14:51] And the agent capability. We democratize it agent capability with a multi-vendor governance. We are we were worried about this because when we we scale the use of agents without governance, we have cows. So, we need to solve this.
[15:07] And we we built a platform integrate with data bricks called at Cielo agents that we can see the traceability, security information, data access, and all the decisions made with the agents.
[15:23] But, now here than the keynote that we saw yesterday, now you have a Unity AI catalog. So, we will help it help you as with this. So, but here is in production, seeing what happened with agents or any decisions
[15:39] made with AI. And uh we have integrated collaboration here. So, all the the agents are shared for the company to use to use the skills or to for usable for any kind of business
[15:55] that makes sense is about the skills that agents are running. And Oh. Okay. And when we enable uh agent capabilities,
[16:11] conversational capabilities, and analytical with data that we can trust, uh here is an example that we are running with both with the uh combining all these capabilities. So, here is an
[16:28] example that a platform that orchestrates and negotiates with our customers just with uh conversational intelligence using WhatsApp. All the the negotiations are made by agents. And the agents make decisions and close
[16:44] the negotiations. And the human are looking for the uh agent behavior. They are looking for the agent behavior and increasing their capabilities. So, here we are combining all these uh conversational, analytics,
[17:01] and agent. Here we will start to work with people, human and agents AI teams. And uh when we start to to work with people, enablement uh
[17:18] to work with AI um teams. Um format for human and agents. We start to we need to change the future, too. Here is is not human versus AI. Is human
[17:34] working with AI, expanding the capabilities for the human. So, we need to start a program for all the company to start to think different. We we we make a program called a
[17:51] brilliant futures that we enable all the teams, business teams, operation teams to work with AI building agents using that those capabilities and to scale problems different than just talk or chat with some AI model. And we start to
[18:08] enable for all the company human and AI teams. And what was the lessons learned? Well, um two years working in a foundation and migrating all the technologies was not a
[18:24] big bang. We work with coexistence with different environments. Um we learned that transformation is a decision, not a tool. Um we need to work with AI as a
[18:39] capability, not isolated projects. So, we thinking analytical, conversational, and an agent for everyday capability for all the teams, not just for the tech team or for the data team. Uh method, base,
[18:54] and knowledge was three pillars to build this transformation. And when we did that, we had to be disciplined to execute, to to teach all the the business person
[19:10] to uh making the the their decisions inside the new foundation, too. And um AI only scale when foundation is ready. We need to trust in our foundation. We
[19:25] need to trust in our decision and the data quality that that is inside the foundation. And here is um presentation that we change the culture of the company. It was not just a technical. The company
[19:41] starts to build and to think how to solve uh problems with AI. How to solve problems thinking first in data. So, that's the main difference that we did here. So, we enable the company to solve their problems with AI here and with
[19:58] security, try uh traceability, and privacy things. All that we need uh to to trust in our decisions, too. So,
[20:14] uh here well, to just to end here, I think the the future was not how we can transform AI like technology. The future is like we work with AI as a capability inside the business, inside
[20:30] the decisions to accelerate our decisions and to be uh to to transform the experience for for our customers, too. Okay? So, that is Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.