Rakuten’s Journey from Data Silos to Unified Intelligence

Summary

Rakuten built Databridge on Databricks to connect data silos across its 70+ businesses spanning 30+ countries without physically moving underlying data by default, serving over 2.1 billion global members.
The platform combines Unity Catalog, Lakehouse Federation, and a custom Data Access Management Service (DAMS) on Databricks Apps to enable governed, zero-copy federated access across multi-cloud and hybrid environments at enterprise scale.
This video covers the vision and architecture behind Databridge, including design adjustments that made federation workable across over 33,000 data connections and lessons from operating governance, performance, and audit in production.

Rakuten’s Journey from Data Silos to Unified Intelligence

Rakuten Group's 70+ businesses span e-commerce, fintech, mobile, and more across 30+ countries and regions, serving 2.1B+ global members and ¥48.7T (approx. $300B) global gross transaction value in 2025—one of the world's largest interconnected digital ecosystems. As the Group accelerates AI-driven products, copy-heavy integration of data in multiple silos across cloud and on-premise environments makes gaining real-time insights challenging. Rakuten built Databridge on Databricks to connect those silos into a single, searchable, secure hub—without moving underlying data by default. By combining Unity Catalog, Lakehouse Federation, and DAMS (Data Access Management Service) on Databricks Apps, the Group is shifting from physical copying to governed, zero-copy federated access at enterprise scale. this video shares the vision and architecture behind Databridge: how Rakuten unified discovery, access, and compliance across multi-cloud and hybrid environments, the design adjustments that made federation workable, and lessons from operating governance, performance, and audit in production.
Talk By: Prakash Balraj, Engineering Manager, Rakuten ; Taku Okoshi, Executive Officer, Director, Rakuten Group, Inc. ;

Chapters

FAQs

What is Rakuten Databridge and why was it built?

Databridge is Rakuten's unified data platform on the Databricks Data and AI platform, built to connect data silos across 70+ businesses and 30+ countries without physically copying data by default. It addresses the challenge of gaining real-time insights from fragmented multi-cloud and on-premise environments serving over 2.1 billion members.

How does Rakuten use zero-copy federation on Databricks?

Rakuten uses Databricks Lakehouse Federation to provide governed federated access to data across cloud and on-premise systems without requiring physical data movement. This shifts the Group's approach from copy-heavy integration to zero-copy access, reducing duplication and improving governance.

What is DAMS and how does it control data access at Rakuten?

DAMS, Rakuten's Data Access Management Service, is a purpose-based access control layer built on Databricks Apps that governs who can access which data and for what purpose. It enables cross-business data sharing at enterprise scale while respecting the regulatory barriers to data sharing present in Japan.

What results did Rakuten achieve with Databridge on Databricks?

Rakuten achieved a dramatic reduction in data access time by shifting from copy-heavy integration to governed federated access. The architecture supports over 33,000 data connections and is being extended with a roadmap of six initiatives including an agentic supervisor system.

Full transcript

[00:08] Let us start our presentation. So first let me introduce myself. My name is Taku Okoshi. Uh I came from Japan. So uh we are belongs to the Arctang group. Then uh today uh we really want to introduce
[00:20] our journey from our data silo to the unified intelligence and I will cover the overall our data strategy first then after that so praash baraji so our data
[00:35] engineering lead will touch upon the detail of the technology.
[00:48] So I believe uh here so um I I don't know how many of you know about Raken. So Raken is the uh kind of so uh Japanese internet company and so we are covering over 30 countries across all
[01:02] over the world. uh in uh here so US San Francisco uh we are running lacten.com uh as a services then uh react.com provide the uh kind of so cashback
[01:14] services uh for the uh internet uh shopping and uh real uh how can say real store shopping too and uh our main location actually is Japan uh our
[01:28] headquart head headquarter is also located in Japan in Japan Then we have the various type of businesses like e-commerce or e-commerce business size is same as the Amazon Japan level. Then
[01:42] we also have a number one travel booking agency services we call react and travel. Then uh we also have multiple financial services like a bank and security insurance. So we are so a
[01:56] number one credit card issuer in Japan market. So uh recently our uh the number of our card which we issue reached to the 30 million then so the together with
[02:09] these services we are building ractton ecosystem there. So number of services in racten ecosystem are already re70. Then from data strategy angle our
[02:22] mission is the uh one customer uh one intelligence through our ecosystem we can gather the so a lot of user behavior like so user purchase history in the
[02:35] e-commerce so user purchase history through the credit card. So recently we are also running this rect mobile services. So this is kind of telecom services. uh through this rectang uh we
[02:48] also can understand uh what our user is doing on on the so mobile devices such a very rich and variety of this user behavior is the our asset then so based
[03:01] on that so we try to uh build our hyperpersonalization capability then so again so our theme is so one customer and one intelligence
[03:13] so this is one example so bas based on our data asset. So since we can understand the user very well. So based on our data uh we can fully utilize such a data asset for the internet
[03:26] advertisement targeting. So we also can provide so intelligence marketing capability to our client uh B2B client. Then we also fully utilize such
[03:39] capability uh for the uh the defensive side. For example, uh we understand bad user behavior and so based on bad user behavior uh we can detect the fraud and
[03:51] uh we also can provide the risk management capability for our internal services and for the external services. So the uh our mission is like this. It sounds very beautiful and very nice but
[04:06] actual situation is uh real situation is very tough. So uh to unify the data so we have a lot of challenges. So one challenge is the uh fragmented data
[04:18] architecture. So uh actually so each 70 services in react and group are using completely different data architecture. Service A using the Google cloud.
[04:30] Service B are using the so aure. Uh some services are using the so AWS and some service own their data warehouse on the private crowd. Then uh since each
[04:44] service develop their own data warehouse nobody care about entire group view. So eventually the uh our number of the data warehouse leads to the over 100. So it's so complicated. So among research data
[04:58] warehouse so many user try to connect the so data. So that's why data pipeline among resour data warehouse also uh become bigger and bigger. There's a
[05:10] number of data connection already leads to this 33k. So you can understand how complicated data is from data governance point of view and data utilization point of view.
[05:22] This is one of the our challenge. The second challenge is regulation uh uh and restriction. Uh as I told you so ecosystem uh include the financial
[05:35] services bank has their own regulation security has their own regulation this credit card insurance too. even the e-commerce. So there are many many act
[05:47] which we have to follow and since this ecosystem is quite unique in Japan the Japanese government keeps their eyes on the what we are doing.
[06:00] fairness uh in the each business industry. So then uh because of this regulation uh when uh we consider this data utilization uh for example someone asks say hey I
[06:13] want to combine service A data and service B data is do uh some how can say targeting algorithm so uh at that time so our side has to check this so their
[06:26] application is pretty much follow uh the uh guideline and the regulation or not. Then so since regulation is so much complicated so there are many many
[06:38] various type of the regulation. So the this data governance process tend to take us longer time. So in some specific case so one application takes or two or
[06:51] three months to get approval. So the under such a situation so we cannot achieve for example one customer one intelligence right. So it takes so several years or decade. So that to to
[07:06] avoid such a situation we came up this three strategy in our data uh as a data our as our data strategy. One is this unified discovery. So nobody uh know the entire so data set
[07:22] in our rectang group. So even so they are the data specialist we have to provide so unified data discovery capability then uh to avoid any
[07:34] regulation uh violation uh we try to minimize or uh we shouldn't have this copy data for example in the banking or
[07:46] credit card businesses government ask us hey you should put us data in this location you shouldn't shouldn't copy at to this location. So there are many many such type of regulation to avoid uh to
[08:01] overcome such rule and regulation. So zero copy access is more better way then again so we have a multicrowoud uh situation uh Google Azure AWS
[08:15] on pre data platform should have this hybrid hybrid flexibility. So we have to connect the data uh uh across this all crowd vendor including a
[08:27] private crowd. Uh to align with these three strategy uh we recently launched this new internal products we call uh reactant data bridge. So data bridge has
[08:41] two meaning. One is the uh data connection uh across the or multiple business domain like a bridge. The second uh meaning is so this datab bridge fully supported by the datab
[08:55] bricks technology as you already know headquarter of the datab bricks located in the San Francisco San Francisco has a many many beautiful bridge including golden gates bridge so that's why we uh
[09:08] picked this bridge as a word of the uh uh service name then uh we call this data bridge then this data bridge capability uh has a three capability one again so is the
[09:23] data discovery and understanding capability uh through the unity cut off then uh we also apply the same governance rule and security rule uh with unity cut capability then datab
[09:37] bricks support our kind of data connection uh parts with federation capability so these three main uh uh function really align with our uh three
[09:50] strategy. We just launched this data bridge recently. Then uh once uh this is kind of a shortterm goal which we really want to achieve in 2026
[10:04] and uh one use case of data bridge is so improvement improving this our advertisement businesses. So once we migrate uh integrate all data so we can
[10:16] expect so 0.3 or 0.5 uplift on our advertisement revenue number sounds little bit small but our actual so advertisement business size is quite
[10:28] big. So even improvement size is small. So uh monetary value is so big. Then uh uh we also can expect uh many data management data governance operation uh
[10:42] cost reduction. So we aim to achieve this 80% reduction. Then as I told you at the beginning so this is the kind of shorterterm goal. So in 2027 2028 so we
[10:55] fully expand such data bridge capability across react and group so together with the datab brick support and so uh we can have some more merit so business merit and the tech merit uh in the future. So
[11:10] uh this is the kind of overall our locking group strategy and our landscape. So from here so Baraji will touch up on this more detail part of this technology. So let me toss to him
[11:23] and over to you Baraji. >> Thank you Akosan for the insights. Now let's look at the technical implementation of how we implemented uh data bridge in our datab bricks platform. So as senior engineers and
[11:37] architects whenever uh business say uh they want to uh build a unified dashboard or enterprise grade model um our engineering reflexes will say let's move all the data into a single physical
[11:49] repository and build massive ETL pipelines and move pabytes of data into your common uh physical repository. In any enterprise uh you will have a transaction of e-commerce service DB or
[12:01] ultra secure fintech service DB and you will have unstructured data either in Azure or in GCS and you'll have your legacy pipelines running in your on-remise platforms. Uh moving all this
[12:13] data into a single physical repository uh it's not only uh expensive it will introduce uh compliance risk uh compliance risk and an massive engineering overhead.
[12:26] So uh the architecture that we are looking here is the universal gateway architecture concept conceptually as a data bridge platform. It is powered by unity catalog and leos federation through zero copy federated access. Uh
[12:38] whenever the downstream systems like cross service BI or enterprise AI model queries this the query engine will push us push down the data to the corresponding service where it where it is appropriate.
[12:54] So this is the highle architecture of our datab bridge platform. So our datab bridge uh is hosted in uh datab bricks azure. Um so bigquery is one of the uh largest external system that we have
[13:06] right now and we have some of our legacy pipelines running in onremise and we also have some of our fintech services running as a native delta tables in the Azure datab bricks itself. So here Unity catalog acts as an abstraction for our
[13:20] uh governance here. So for bitquery and on-remise systems we use leakos federation uh to register those into the unity catalog and create as a foreign catalog and for our datab bricks uh
[13:33] which is already there for our uh fintex services we just bind it into our uh workspace so that the user gets a unified view for the discovery.
[13:46] So from here on we will look into our uh Bitquery federation strategy because Bitquery is one of the largest uh external system for us. So we will from here on we will look through the Bitquery part. So within our GCP environment uh data is isolated into
[13:58] product specific data pro projects each secured by its own VPC service control parameter to bridge this securely with Azure without public internet exposure. We route all federated uh access through
[14:10] a single uh compute project in GCP side by whitelisting our data bricks outborn ips uh at the GCP parameter. So inside this compute project we have a product specific service accounts through which
[14:23] we handle the data access for the corresponding data projects and we whit list those service accounts and net ips in the data project as well. So with that the security parameter active the uh now let's look into the uh four steps
[14:36] how our data flows from uh Azure to the GCP side. So on the Azure side when user queries the data via data brick SQL or spark uh unity catalog uh accepts a query and translates the dialects and
[14:49] passes it to the cloud boundary across the boundary to the GCP side since we have already whitelisted the net ips in our bigquery side. our compute project uh receives that request and then since
[15:01] the service accounts and then the IPs are already whitelisted it passes down the query to the data project and then uh all the predict since the predicate push down is pushed to the bigquery so it executes locally in the bigquery and only the final filtered uh data set is
[15:14] passed over the uh internet to azure passed over the egress to the Azure environment.
[15:27] So uh till now we have seen the how the federation works but we also had a bottleneck of governance. Uh historically our governance is a major bottleneck as Okashan mentioned. Uh because each domain has their own forms
[15:39] own process and own siloed approvers. Uh so the average weight time is more than a month and in an era of a making a power decisions this has become a liability for us. So to solve this we built uh DAMS data access management uh
[15:54] which is a single unified self-service platform natively deployed on uh using data bricks apps we have built. So it provides a dual dual uh layered guardrail uh combining role-based
[16:06] structure with purpose-based access. First the roles are tightly aligned with our organizational chart like rakutan group rautan mobile etc. Within these uh roles, our chief data officer
[16:18] predefineses valid purpose of use like analysis or marketing. Each purpose is aligned with our DCR which is backed by the legacy and uh compliance part. So the access is never grant granted to
[16:31] your table forever. It is securely tied to your business purpose with active compliance backing. So uh whenever the user applies the corresponding role tied to his organization uh since if he subscribes
[16:44] to that particular role he uh there is a Jira back system which rises a ticket to the uh system and then once the approver uh will be his manager once he approves our backend system will provide the
[16:57] access to the corresponding uh list of groups that is granted. So we did not trade compliance for speed. Uh so we have uh kind of like every access request is tracked and every access
[17:10] grant is also locked and all the approvals is also locked here. So we we have built uh uh audit dashboards mainly for our data owners and also for the
[17:22] admins and the financials uh financial teams. So access audit dashboard is primarily for the data owners to have a transparent view of catalog usage and access. So whenever there is a
[17:34] permission change or the usage uh new permission is applied or the permissions are deleted, we send a real-time alerts to the data owners to ensure the compliance and we also have a dashboard view for data downloads and usage. The
[17:48] we we track the usage mainly for optimizing our uh resources and we also uh track all the downloads to prevent any data breach data breach issues and
[18:00] we also have a cost tracking dashboard as in federation the cost is spread across Azure and GCP uh combining into a single dashboard helps our financial team to corresponding chargebacks.
[18:15] So now let's look at some of the challenges that we faced and how we ensured using some strict engineering policies. How do we how we have fixed we'll we'll see from here on. So first challenge is the workspace strategy. Uh so when architecting our data bridge topology we faced a fundamental decision
[18:28] on how do we support diverse enterprise user profiles without triggering workspace problems. So basically minting a new workspace for every team or minor project create create a inconsistent
[18:41] configurations and fragmented storage endpoints and very heavy operations are involved as we need to whitelist the projects uh IPs datab bricks IPs within our corresponding uh GCP projects.
[18:54] So uh if you look at the use case uh we have looked at the use case of our all our platform users. So nearly 80% of users fit the adoc or small scale profile users. So we have del deliberately uh had a shad first
[19:07] strategy and we in our workspace we have enabled all the strict permission entitlements and then cluster policies and then we are onboarded them to the shared workspaces. So once the access is
[19:20] granted we are dams so the user can quickly start using and accessing the platform and do the analytics. But uh if certain uh teams have a very strict isolation uh network isolations uh due to the organization policy, we create
[19:33] dedicated workspace and setup for them as well. Now let's look at some of the federation performance. uh since we are using uh query federation uh the uh access has to
[19:47] go through the bitquery storage and the data moves over the egress. Of course we will have some kind of performance uh issues as well while doing the federation. So we have do done some kind of like engineering practices to
[20:00] overcome these issues. So one issue is like moving massive data sets across the cloud since we are transferring the data from GCP to Azure over egress. So we ensure the maxim uh we ensure the
[20:13] maximize predicate push down is applied. So datab bricks uh leos federation connector inbuilt it uh works it supports the predicate push down. So we uh uses uh
[20:26] maxim predicate push one over the web and we have enabled some kind of uh watchd dogss so that like we whenever the data is transferred over internet and coming we have like not more than
[20:38] two 100 million records are getting transferred over internet. So that we have to and also some kind of there are some limitations like uh udfs are not supported if there is a suicide filter is applied. So those things we uh
[20:50] followed a sick practices that there and uh we also uh had our driver memory issues when the full datas are like full scan is applied and the data is transferred. So we uh encourage our
[21:03] users to use proper uh filters in the query uh to have to have all the filter data is pushed back to the Azure side. Yeah. So and the final point so the full
[21:17] table scans. So we also have some kind of like workspace skills that we have set. So like whenever user uses the natural language uh to build a query so we ensure we have kept a proper skill
[21:29] files like uh whenever the natural language query is coming up. So so it it applies a proper filter proper uh uh site join. So those kind of filters are mentioned in the skills so that like the
[21:42] natural language query has a proper query. So when the user executes also it's uh will not have an issue. So u here's a for the billing building part. Uh so as I said like the cost is
[21:56] split across Azure and GCP and uh currently datab bricks does not have a consolidated view for the federation itself. We have we can have the cost for the data bricks but we cannot have the combined cost. So we have built a custom
[22:09] solution to get the complete federated cost in a dashboard. So what you have did is like as we are using a shared workspace like we have created individual clusters. So basically the
[22:21] push down ts are not applicable for GCP right now. So what we build is like currently for allp purpose cluster whenever we are giving for the users we have added custom labels like uh since bitquery uses a storage API storage
[22:35] spark connector we used the um bitquery job labels which will push down to GCP and uh we kind of like filter out based on the tax that labels that we set we calculate on the GCP side and then move
[22:47] this data back to the Azure to the delta table. So we build a custom billing table where it uses the Azure cost for the including. So in Azure also as you know like when the queries are executing you have a separate VM cost if you're
[23:00] using uh classic resources. So we calculate all the VM cost from the Azure storage and then the corresponding data bricks DBU cost and then the GCP cost is combined and on top of uh on overall we
[23:13] build a cost dashboard for the unified one. Of course like uh in the data bricks currently the push down labels can set only on the allpurpose cluster the SQL varrows and serverless we cannot have the push down labels and we have
[23:26] raised a feature request for that part. So uh what it means to the business after uh delivering uh data bridge. So as we have implemented dams uh we have
[23:39] seen a dramatic drop in that uh time to access and this immediate access immediate access accelerates our decision speed uh our teams are now making vision uh business decision forex faster and uh uh also as part of
[23:53] operational effort since we are not kind of like duplicating the data we don't have to build any data pipelines or like creating multiple workspaces for uh each decision each small scale. So that will be 83% reduction as the overall goal
[24:06] that we are targeting. Yeah. So um so let's look at what what we have done and what what's up for our road map. So we don't just build a bridge to solve
[24:18] yesterday's access problem. So we built a foundation for tomorrow's autonomous AI. So the visual of the bridge as you can see perfectly illustrates our current phases fully built uh solid and supporting live traffic but it's
[24:30] explicitly designed to uh scale it further. So currently we have deployed onboarded 35 plus data catalogs and enabled federation for them and then we built a access management DAMS portal for users. So what's up for our uh next
[24:46] frontier? So it's like we plan to uh onboard all 100 plus rakutan data warehouses and then also onboard near realtime pipelines for the making personalization and ads and also make our platform completely autonomous by
[24:59] having uh agentic solutions. So uh this slide illustrates uh how we are transforming from a pedration platform to a complete data value chain.
[25:11] So we have a six initiative in our road map. So as you see the data bridge platform will have the data coming from all 70 plus services in future and we will have a standardized data connection for all our BIS marketing ads and all
[25:25] our AI agents and we also have our internal R memory LLM service sorry internal LM service uh basically that service will enable a applications to
[25:38] persist the contextual memories of our user conversation. So we will feed the uh data. Sorry about that.
[25:54] Okay. Uh so basically we will feed our data to the memory servers and uh we will have a near realtime pipelines. uh currently as you know the with all batch pipelines that we are federating in future we will support the near realtime pipelines federation as well uh for uh
[26:08] improving for ads and personalization and we plan to add the unstructured data as well oops I think there's some issue with the maybe uh we plan to add the unstructured data like shareepoint data
[26:21] or any documents related data uh for our AI agent specifically and also we will uh add the agentic operations layer like data ops agent and PHOPS agent which I'll cover in the coming slides.
[26:37] So to successfully scale the six future initiatives we just mapped out our underlying storage layer also must evolve. So uh currently our external systems have different storage layers like uh BigQuery storage or it can be a
[26:49] multiple different storage formats will be there. So we need to adopt to the open table formats which will uh give us a universal standard without format translation. So, so as you know like uh currently uh each analytical engines
[27:02] will use their native per engine table formats. So when they are translating their formats it takes it will impact our uh performance. So only the partial predicate push down can happen over the
[27:14] network overhead. So and also whenever a new engine comes there will be a new integration mode. So as part of our federation 2.0 so we wanted to introduce open table formats for all our source systems. uh so that like it will be
[27:27] unified it will have a universal format and maximum predicate push down will over so we can reduce the network transfer and also any new engine comes it will be just plug and play
[27:41] uh so this is our AI powered operations for our future so currently we are collecting all the logs uh from the data brick system and also the we also will have many data pipelines or any multiple pipelines that will be running in data
[27:53] bricks we collect the logs so data Ops agent is one of the agent that we uh building. So for uh pipeline auto repair. So whenever the job fails the agent will uh it is a multi- aent system which will have a centralized super uh
[28:06] supervisor system supervisor agent uh which will send the to the analysis agent and then there will be a recovery agent to recover the failures with the knowledge. It will refer to the knowledge base and then uh recover the
[28:18] pip data pipelines. So and also it will have a self-learning capability to uh learn the new errors and update the knowledge base accordingly. And we also have a phops agent that we are developing. So mainly to minimize the overprovisioning and under utilization
[28:31] of computes based on the uh s uh users usage for the corresponding clusters. So it gives the right sizing recommendations and also to detect any anomaly detections of the cost like if there is any spike.
[28:48] Yeah. So as we are coming close to the sessions, I wanted to uh emphasize that the datab bridge platform is uh uh built to support our Rakutan's next decade of AI and innovation. Um from day one, our architecture was strictly divine uh
[29:01] driven by our corporate philosophy, one customer, one intelligence. U so we built an adaptive ecosystem where tools server strategy rather than letting uh tools limitations dictate our strategy. So we built our frictionless governance
[29:14] uh by combining pedration and dams and by introducing all our uh agentic system we have a self priming ecosystem. So we built a data engine that will safely securely and autonomously power rauten
[29:27] into the future. Thank you. Uh thank you everyone.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.