Modernizing Data Governance at First Thai Bank to Prepare for an AI-First Future

Summary

Siam Commercial Bank (SCB), Thailand's first bank founded in 1907, restructured into a holding company called SCBX in 2021, expanding from one entity to 12 subsidiaries and creating a complex data governance challenge across distributed data silos.
SCB deployed Databricks and Unity Catalog to unify governance across 10,000+ data assets from 20+ business units, building a data quality rules engine that reduced processing time from 60 hours to 6 hours while achieving a 15x reduction in licensing and compute costs.
To solve the problem of metadata discovery taking weeks, SCB built a Genie-powered 'Chat with your Metadata' application on Unity Catalog that enables regulators and data scientists to find PII attributes and data lineage in seconds rather than weeks.

Modernizing Data Governance at First Thai Bank to Prepare for an AI-First Future

Founded in 1907, Siam Commercial Bank (SCB) is the first Thai bank. In September 2021, SCB embarked on a landmark transformation, restructuring into a holding company model under SCBX. This strategic shift has since driven the group's expansion to 13 core subsidiaries today, significantly increasing the complexity of managing data governance, data security, and data quality across the group. Today, SCB runs its enterprise data platform on Databricks. In this video, we will highlight how SCB leverages Databricks and Unity Catalog to scale enterprise‑wide data governance and data quality. We will also demonstrate "Chat with your Metadata", leveraging Genie for enterprise-wide metadata discovery on the platform.
Talk By: Sohum Sachathamakul, Solutions Architect, Databricks

Chapters

FAQs

What governance challenges did SCBX face after its 2021 transformation?

When SCB restructured into the SCBX holding company in 2021, it went from one bank to 12 separate companies, each with its own data infrastructure and no standardized way to share or govern data across entities. This created 12 data silos with inconsistent quality, security, and metadata standards—a challenge that had to be solved before the group could pursue its AI-first strategy.

How did Databricks and Unity Catalog improve SCB's data governance?

SCB migrated 10,000+ data assets from 20+ business units onto Databricks with Unity Catalog as the governance layer. A data quality pipeline with a rules engine and scoring system reduced quality processing time from 60 hours to 6 hours, while consolidated licensing and compute efficiency delivered a 15x cost reduction compared to their previous approach.

What is the 'Chat with your Metadata' application and how does it work?

The 'Chat with your Metadata' application is a Genie-powered tool built on top of Unity Catalog metadata tables that allows users to ask natural-language questions about the data platform. The video demonstrates a regulator query that finds all 1,009 PII attributes in seconds, and a data scientist use case where the tool locates a customer ID field across 48 different tables—tasks that previously took weeks to complete manually.

What were SCB's key metrics after deploying Databricks for data governance?

After deploying Databricks and Unity Catalog, SCB achieved a data quality processing time reduction from 60 hours to 6 hours through parameterized pipelines, a 15x reduction in licensing and compute costs, and the ability to discover metadata in seconds rather than weeks. These results supported the bank's broader AI-first strategy across its 25 million customers and 18.5 million daily active app users.

Full transcript

[00:07] So, my name is Soham. I'm going to be sharing a case study where I work with a client in Thailand, the Thai bank. Um but you know, just from this morning session, like you know, what we heard is
[00:19] that AI doesn't have an intelligence problem, right? And it has a context problem. And data governance is actually the core of the context and how context is shared with, you know, all the AI companies and
[00:32] AI projects. So, um in the banking industry specifically, right? We know that many banks are still on their journey to unify data in their data platform and set up the governance
[00:45] side of things. So, we kind of need to move into two motions side by side together to enable AI for, you know, banks. Um just a forward-looking statement here, quite general. This is
[00:57] informational based. And your feedback will help the session, help me. So, please do your feedback after the session is on the app.
[01:10] So, you know, just to just to jump in um the you know, SCB is a Thai bank, Siam Commercial Bank, and they have partnered with Databricks in the past couple years to unify their data platform and their
[01:23] data governance. So, today I'll be sharing like a story that we worked together around data governance and how they're actually thinking about preparing for the next steps of AI in the AI-first future.
[01:38] So, just to give you a bit of context on SCB, right? It is a 120-year-old institution. So, it is a really old bank um formed by the royal family in Thailand. >>
[01:50] 120 years of long history, but you know, the key key points are they went public listing in 1976. Um they had to go through the Asian financial crisis, which was a big uh you
[02:03] know had some big financial impacts across Asia. Um they formed a group called SCBX. This is where they broke out from a single company into 12 uh different companies,
[02:17] right? So, I'll talk a bit about that. And the impacts of data and AI. And lastly, like 2023 onwards, they're really focused on their digital and AI, right? They had this vision of becoming an AI-first bank.
[02:31] So, just a couple key facts. It is a first bank in Thailand. It has around 3 and 1/2 trillion Thai baht of assets, which is around 100 billion US dollars. Um most of the users are digitally uh
[02:45] enabled. So, they're you know, out of 25 million users are all digital enabled, where 18 and 1/2 of them are actually using their app um on a the uh on a daily basis. Um 81%
[03:00] MAU. So, a lot of digital enablement happening with the the bank, right? That's basically the the gist of it. And it's also ranked the top 10% of global banks in terms of banking sustainability.
[03:12] Okay? So, double clicking to 2022. So, what happened? They you know, broke up into um a set of companies, where the first generation is actually SCB, the bank
[03:25] itself, right? The core business. And they split out, you know, their other businesses such as their credit card business, their auto loan business, um into what they call generation two. And in generation three is their
[03:37] platform and technology businesses, like their investment platform called InnovestX, their VC firms, their token uh companies, and so on so forth. And that was all enabled by two
[03:50] foundational companies called TechX and DataX, which were basically their technology enablers. So, what this actually means is, "Hey, now like we cannot look at data governance just in one company. There's actually 12
[04:03] different companies all having 12 different technologies, data stack, data silos. How do we unify and how do we govern it together, right?" That was the problem statement from 2022 for us.
[04:18] And this, you know, there's a couple of key challenges that they wanted to solve. They wanted to remove data silos, right? Across the bank, which held most of the data, but also across the other companies.
[04:31] They wanted an open and simplified architecture, no vendor lock-in, you know, no spending on expensive license costs. They wanted to centralize data governance and quality. And also integrate with a core banking
[04:44] system. Whereas on the group level, the 12 companies had challenges around how do we standardize data across the group? Right? What does a customer mean for us? Is a customer in a bank versus customer
[04:57] in your loan applications and credit cards the same customer? How do we unify data standards? How do we securely share data across the group as well? And how do we, you know, build this data
[05:09] community around the bank? So, in 2022 onwards, we partnered with them and they chose Databricks platform. And mainly, you know, to unify their
[05:21] data platform, but two key technologies were there to enable their data governance and sharing, which was Unity Catalog and Delta Sharing. This actually led to, you know, efficient data sharing between the
[05:34] companies. It actually was, you know, Databricks to Databricks sharing was quite easy across tenant, across cloud. It also led the standardization of their data, right? Moved away from
[05:47] separate data lake, data warehouse, and data systems into one place, the lakehouse. It had everything was on Unity Catalog, and also they were future-proofing the architecture for AI,
[06:01] right ahead of time as well. So, we actually partnered with them from 2022, we started off partnering with them on their enterprise data warehouse modernization.
[06:13] So, they were using a a legacy technology, and they wanted to migrate over 10,000 plus jobs into Databricks. So, that was a where we kind of first did a big data
[06:25] migration project. 2023 was all about driving the machine learning, or back then the AI use cases, right? All of the personalization system, recommendation system, fraud detection,
[06:39] hyper-personalized All of these innovations were like 2023 focus on driving innovation from the data. 2024 onward, there was another partnership that we struck with them.
[06:51] This is to unify the entire data stack of the bank into what we call the unified data platform, UDP. So, this is taking the, you know, enterprise data warehouse, the
[07:03] structured data, as well as like the data lake plus data warehouse into one platform called UDP. At the same time, the bank was also modernizing the core banking system,
[07:15] which, as you guys know, it's a really long transformation journey, right? So, we needed to operate in a way to pull in the data from both new and old core banking systems into the platform.
[07:29] And then there was also the group data zone is where all of them subsidiaries started to share data using Delta Sharing. So, this all happened like 2024, 2025. And this year, the focus is around how
[07:42] all the data is in the platform is on the lakehouse. How do we enable business users using like a citizen data platform? How do we enable How do we bring in more data assets? It doesn't cover 100%.
[07:54] There's still data silos, small specs of legacy systems in the bank. How do we actually unify them in the same platform, right? So, it's an ongoing journey. How do we prepare the data for GenAI
[08:07] projects, right? That's the focus around this year, 2026. And the architecture looks something like this. It's a simplified architecture, but on the the key things is on the left-hand
[08:20] and the right-hand side, right? So, on the left-hand side, they were actually pulling data from on-premise systems. We were pulling the core banking, both the old and the new data
[08:32] into Databricks. They're across cloud sharing as well, and they had a hive metastore as the current cataloging tool. We brought that in
[08:44] using, you know, a basically native Databricks capabilities, as well as Azure Data Factory for on-premise to cloud sort of ingestion, and leveraged the medallion architecture
[08:58] at the bottom. You see bronze, silver, gold. They call it raw, persist, curate. It's the same thing. And they leveraged on the right-hand side is is where they actually use uh Delta sharing to share data across the
[09:11] group. They enabled 20 plus different business units with Unity Catalog access control. That was a big big migration, and also supporting the third-party BI
[09:24] tools like Power BI and SAP Business Objects, SAP BO. In this migration, we didn't want to break downstream, right? So, we just needed to support um, BAU as well of the
[09:38] 20 plus business units using the platform. So, this is good. This is live. Now, we have all the data in one place. At the same time, they were also doing a data quality and data governance
[09:51] migration by leveraging Databricks platform. So, in the next section, I will actually cover a bit on that. So, we'll talk about how they modernize their data governance and
[10:04] data quality the as well in this migration. Okay? This is their data governance framework. So, the goal was to become a data-driven
[10:16] organization. Behind that, there needs to be a strong data vision and data strategy, data governance strategy. And this was supported by four key pillars, which were the data governance policy and standards,
[10:29] stewardship and accountabilities, data quality management, and metadata management. And that needed to comply with all of the privacy and security requirements of the bank
[10:41] and the local regulations, as well as all of the risk management and compliance management. So, in this section, I'll talk a bit about their data quality uh journey as
[10:53] well. The data quality framework was look something like this. On the left-hand side, you had the unified data platform data feeding into
[11:06] this uh pipeline. We uh pull in data from like raw process and curate, and depending on the critical data elements, we applied a series of uh uh uh layers.
[11:19] The first one is the data preparation layer, so pulling the data, preparing the data. Second is a scoring, so you run your data quality rules on top of the critical data elements. That will lead to a successful set of
[11:33] results and unsuccessful set of results which will then get exposed in what they call a scorecard zone. All right, so this scorecard zone covered the key data quality dimensions
[11:46] you know, listed here, um, which are quite general across industry. You have like five or six dimensions here. And at the bottom, everything is on Unity Catalog. There weren't any,
[11:59] um, you know, in the previous state, they were actually removing it outside of the governance into another platform and doing the data quality there. So, in this framework, they actually kept it all on UC um, and then used
[12:12] Delta Sharing to expose the scorecard to Power BI. What this also enabled is, um, the relevant uh, stakeholders, for example, the data
[12:25] stewards, data owners, they were able to view the dashboard on Power BI, view the scorecards, and also able to get permission to go into Unity Catalog and actually query against the data and understand where the data
[12:38] quality is is happening, right? Yeah, so their raw, process, curate is basically bronze, silver, gold.
[12:57] So, this is the scorecard that was being maintained, you know, for many years now. Um, the first step was not to change the scorecard, you just needed to feed a different source, right? This scorecard actually represents a retail customer segments. So, the way
[13:11] they do it is they have a set of, uh, critical data elements for each retail customer and they apply 77 data quality rules against the groups of the customers
[13:23] which then, you know, you get this big complex scorecard with all these different percentage scores and then that basically will like um get updated and will show you where the data quality issues happening on a retail
[13:36] customer view. There's they have another view of the application layer now. So, a complex data platform like this will
[13:48] have you know, many uh 50 plus different source systems, right? So, we also wanted to see what source system is actually having the issues. So, this view is on
[14:00] the application or like source system layer. Each application has a set of tables that are ingested into the platform, whether it's in raw persistent curate, right? Then, you would run your
[14:13] daily, weekly, or monthly data quality rules against these tables uh for each of the application, which will then also reveal uh scoring and overall scoring as well as a
[14:26] specific by critical data element scoring. And then there's a dimension view on the top right there.
[14:39] So, what did they have before? Before, they were not using Databricks for their data quality pipelining. They were using another technology. And basically, it was the the step number one is they're
[14:51] actually pulling data from the data platform into another software. Right? This created uh data copies, data silos, uh bro- broken data governance, all these things.
[15:04] After preparing for each table that comes in, they would also create a rule uh for each of the tables, like a separate notebook for the rules. They would have a data quality score
[15:18] notebook where they would actually process for each table, for each rule. They would run the data quality scores, write it into two tables, which also lived in a separate database system.
[15:31] Which then gets exposed onto their dashboard, which I showed you just now. So, there were a couple of areas they wanted to move out from here, right? Number one, data copy. So, they didn't
[15:43] want to uh have data copies. Um They wanted to move the data quality to the data, not away from the data. Second thing, it was very manual. So,
[15:55] every time there's a new data set to be ingested, then we they needed to create these very bespoke code for each of the tables of the rules. Um so, those were some areas that they
[16:08] wanted to uh improve in this migration as well when they were moving to Databricks. So, this led to something like this, right? You have your source data. You keep the
[16:21] ingestion code. So, there is some pre-pre processing needed to be done. Once that's done, there's a set of configuration file where you define the parameters for validation.
[16:33] You have another file which has all of your rules, which is common and shared across, you know, all the tables. That gets fed into a single Databricks pipeline in parameter like parameter-driven
[16:46] pipeline, which then creates your calculation of your score, your data quality scores, and then writes it to a Unity Catalog table. This is a from end to.
[17:00] Now, if we look at the impact, actually, it seemed like a pretty straightforward data quality pipeline, right? It actually led to significant performance benefits.
[17:12] If we just look at performance alone first. In the first uh chart up there, you have your retail customers. These are monthly jobs they run, and it typically takes 60 hours to run.
[17:25] First step was we tried to just throw it into Databricks. Like, take the code and throw it into Databricks and just run it. That alone actually brought it down by more than 50% uh, performance.
[17:40] Just out of the box. Then the next phase was like, "Hey, let's actually, you know, right size of VMs, pick the right VMs for the right jobs, and you know, have uh, parameter configurations in there as well."
[17:53] Again, that half the the time to run the jobs. And then the last step, which is the current state, was around "How do we actually, um, optimize the flow by having like a
[18:05] staging layer, which actually saved a lot of the time of processing? And also, how do we create a framework that doesn't, um, fail the entire pipeline if there's an error in the jobs?"
[18:18] So, actually, they split this out into parameter driven, uh, data quality pipelines, which is all shared by one single notebook. Yeah? All parameter driven.
[18:30] So, this actually led a huge performance improvement back to 6 hours from 60 hours just by doing this. And this actually, you know, if you look at the second and third chart, there's also even more like
[18:43] performance improvements. For example, the daily jobs was from 8 hours to half an hour to run the daily jobs for this data quality pipeline. So, there this was quite significant performance improvements, right?
[18:58] And if we look at cost as well, so cost there was huge 15 times cost savings. It's not 15%, it's actually 15x cheaper.
[19:10] So, before in their current, uh system, they had all of the implementation costs, licensing costs. They needed a vendor who is a special specialized in that particular tool,
[19:23] that language. And also the cloud compute to run was really significant. Um they decided that they can run this all in PySpark. They don't need any
[19:35] vendors to do to manage this. There's no license cost, of course, when using Databricks. It's all the Databricks cloud compute. And because of the performance improvements, it actually brought down
[19:47] the cost quite significantly. Right? And this led to, you know, all 18 million customers of uh SCB being uh having like 120 data quality
[19:59] rules on each customer. So, this was quite a good impact. Now, before we move to the next session, I just want to highlight like
[20:12] um So, where SCB has up until now, they have data governed in one place on the Unity uh catalog. They had trusted data assets with all their data quality. Now, it's also migrated to Databricks jobs
[20:26] and notebooks and pipelines. And they were thinking how they can um democratize their data as well. So, the current state is every time someone
[20:40] wants to discover what data exists in the bank, it's a lot of manual steps. And they wanted to say, "Hey, why can't we just chat with your metadata?" Or create a tool to enable people to
[20:52] chat with your metadata. So, just to recap. So, currently users search the DGO portal, Data Governance Office portal. They uh have to manually,
[21:07] you know, um update all the data stewards in a monthly email blast. And it was quite a slow manual process for a new project to discover what data is actually there. Because currently you only get you can
[21:20] only see what you have access to. If you don't have access to a particular table, you don't even know it's it exists in the bank. It's a lot of manual back and forth. A lot of uh expertise of the bank and experience
[21:33] needed to actually locate these data assets. So, they wanted to move to a chat with your metadata. Um where users can pass natural language and actually understand what data is
[21:46] there in the bank, right? And they wanted to this to be quite automatic or semi-automatic and didn't want to have a lot of manual steps along the way.
[22:00] So, they built a Genie-based application, which I will demo after this, where if you look on the left-hand side, when there's a business user coming in or a technology user coming in,
[22:13] they can directly go to Genie. Um Genie will read uh from three different metadata tables on Unity Catalog. One is for schema, one is for table
[22:26] descriptions, and one is for column descriptions. So, they will This is their interaction. They basically just need to go to Genie. Behind the scene, Genie actually uses a SQL warehouse to power the compute and
[22:40] uses uh and only focuses on these three tables. Right? So, it doesn't do a huge scan of your entire Uni- Unity Catalog. It actually goes to these three tables and it can give you the response of the
[22:53] the queries that come in. And if we shift from the right going left now, so what the what we need to do, So, there is a metadata administrator who basically manages an Excel. That
[23:08] Excel, you know, gets exported and ingested directly to Unity Catalog. So, there is a ADF ingestion pipeline, which triggers a Databricks job, which then syncs to the three Unity
[23:22] Catalog tables. Yeah. So, there is still a Excel being managed. It's not completely eradicating Excel yet. But this way it kind of automatically syncs to one source of truth, which is on Unity
[23:35] Catalog, the three tables there. So, I will demo uh two things. So, one is around for a data governance office
[23:49] uh or uh someone in the data governance team, what how they can use this Genie application, right? So, the demo is actually done by Khun Tatapon, who cannot make it today, but um he was generous enough to give us a
[24:03] recording. So, let me play that. I think there's audio as well. Just double-check.
[24:20] >> Our team search a data portal one area at a time. Now, they can just talk to it. This is data big one. I open our space chat video metadata. Let me start big. I ask, "What data do we go on and
[24:32] how much?" This is live. I can't do real catalog here. Here it is. 16 business area, more than 20,000 data
[24:46] attribute. Lending is the biggest. Secure and secure loan together holds 6,400. My team keep it in spreadsheet. Genie built it at the second. Let's move to the next question. I asked
[24:59] where does the data live? Okay, the top 10 system holds about the 20% of everything we catalog, a map of
[25:13] the whole bank in one question. Now, a question every government team is clear a regulator asked, where do we keep customer national card ID?
[25:29] Before this work week of work, many team, many email. But what here, Genie search in the whole catalog at once, system by system, data set by the data set, column by the column. No spreadsheet need, no waiting
[25:43] for a monthly email.
[26:04] Yeah, 1009 attribute every place in live. The full list ready for the regulator, week of work in one sentence.
[26:21] >> So just this is one like use case where like you can use this Genie application. Just to recap, so regulator comes and they ask, "Hey, I want to know where you're storing your passport ID or your national ID number in the bank."
[26:34] Can you answer that today? I'm not sure, right? It might take a lot of work, a lot of emails back and forth, a lot of queries to build around getting that simple request. Um
[26:46] and that's fair, but I think now with with this sort of application, they're able to wait a couple minutes and it actually scans the metadata and gives you something that you can export it out and send it to the
[26:59] regulator in minutes. So, let's shift into another persona now, which is the data science team. So, that's this is a data science team that's starting a new AI project or a
[27:11] new machine learning project and they want to explore particular features of their data and how they can use it.
[27:27] >> Now, a different person, me, a data scientist, I'm building a single customer view. The hardest step across 100 of table, what field identify a customer? So, I ask.
[27:40] Normally, this is day of digging at email to govern end. What's Genie work? It search a whole catalog. It check a system by system.
[27:52] Also, after system, 100 of table I have never seen. So, let's check the result together.
[28:16] And here it is. 48 customer ID field across 26 system and it's not just list them. It explain them. My main choice here is customer ID and the tricky one I would never find it alone. The corporate ID, the salesforce ID, the auto finance
[28:28] dealer ID. Now, I go deeper and notice that this is still same conversation. Genie, remember the previously context?
[28:40] I'm modeling the auto loan risk. I need feature, income, day past due installment.
[28:55] Once it pull the exact field from the right system here, already made feature list, our data sets and column and claim remaining for each.
[29:15] All-in-one system, our auto product platform and every field is marked as confidential. So, I know the sensitivity before I touch the data.
[29:29] Now, key takeaway is years of governance knowledge now in sense, a plain English and always the current. That I is how we modernize data governance for
[29:41] the AI first feature. Thank you. >> Yeah, so I know a lot of the demo was blurred because it's actually based on the real data, so it's a little sensitive, but I think the idea is that
[29:54] you're able to do a discovery step without, you know, doing any complex queries, back and forth emails, manual, you know, going up to people and asking the IT team, "Guys, what data do you
[30:06] have?" This can actually be done now using this Genie application. And it actually will enable the data governance team to simplify the work a lot more, right?
[30:18] So, this is um pretty much the main sharing of of today. I do want to give a thanks to the SCB team who was not able to be here today. And this
[30:30] journey of modernizing the platform, putting all the data in one place on Unity Catalog, it actually takes um many, many months of hard work. You can see someone sitting on the floor
[30:42] there trying to debug something. So do you want to give a lot of thanks to the SCB team for giving their trust with us. And if you want to you know stay on and talk to me about this the the journey
[30:55] and how you know we're thinking about the data governance. We're also you can I'll be here but you can also reach out to me on this QR code. I'll give you guys some time back. Thank you so much for joining the
[31:07] presentation.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.