From 2-Hour Data Ingestion to 2 Minutes: SAP Integration with Lakeflow Connect

Summary

Panasonic of North America reduced SAP S/4HANA ingestion time from over 2 hours to just 2 minutes using Lakeflow Connect with Auto Loader and CDC, while also integrating Workday, SharePoint, and Salesforce through native connectors on the Databricks Data and AI platform.
The transformation consolidated eight global business units and hundreds of data sources onto a single governed platform with Unity Catalog, enabling row-level security for sensitive HR data and multi-cloud data sharing through Delta Sharing.
Databricks Genie enabled business users to query integrated data in natural language without writing SQL, completing the shift from a fragmented legacy architecture to a self-service data and AI platform.

From 2-Hour Data Ingestion to 2 Minutes: SAP Integration with Lakeflow Connect

Enterprise data integration traditionally requires manual orchestration, expensive third-party connectors, and complex CDC configurations. Panasonic faced this challenge across eight global business units, managing hundreds of data sources with SAP ingestion times exceeding 2 hours, causing operational bottlenecks and unreliable pipelines. Lakeflow Connect addresses these challenges with native, managed ingestion powered by serverless compute, replacing custom scripts and expensive connectors.
this video explores Panasonic's complete transformation: reducing SAP S/4HANA ingestion from 2 hours to just 2 minutes using Auto Loader and CDC, integrating Workday, SharePoint, and Salesforce through native Lakeflow connectors, and establishing centralized governance with Unity Catalog. You'll learn multi-cloud data sharing patterns with Delta Sharing, row-level security for sensitive HR data, and how Databricks Genie enables business users to query data in natural language without writing SQL.
🤝

Chapters

FAQs

What is Lakeflow Connect and what problem does it solve?

Lakeflow Connect is a native, managed ingestion capability in Databricks that replaces custom scripts and expensive third-party connectors with serverless, automated pipelines for sources like SAP, Workday, SharePoint, and Salesforce. It addresses challenges enterprises face with complex CDC configurations, manual orchestration, and unreliable legacy ingestion pipelines.

How did Panasonic reduce SAP ingestion time from 2 hours to 2 minutes?

Panasonic used Lakeflow Connect with Auto Loader and change data capture to replace a slow, custom SAP ingestion pipeline. The reduction in ingestion time from over 2 hours to 2 minutes eliminated an operational bottleneck that had prevented timely access to SAP S/4HANA data across their business units.

How does Panasonic govern sensitive HR data from Workday on Databricks?

Workday data ingested through Lakeflow Connect is governed in Unity Catalog with row-level security applied to sensitive HR records, ensuring that access to employee data is controlled at the data layer. This governance approach allows HR data to be shared across the organization while enforcing privacy boundaries.

How does Genie fit into Panasonic's new data architecture?

Databricks Genie enables Panasonic business users to query integrated data from SAP, Workday, SharePoint, and other sources using natural language without writing SQL. This self-service capability was a key business outcome of the transformation, making the unified data platform accessible to non-technical users across Panasonic's global business units.

Full transcript

[00:08] All right. From 2 hours to 2 minutes. That's how long Panasonic's largest SAP table took to ingest before and after they standardize on Databricks. The story is bigger than one table. Eight business units, hundreds of data
[00:24] sources, SAP, Workday, SharePoint. Whole set of legacy architecture challenges and a decision to centralize their data in their strategy on one platform.
[00:41] Good afternoon, everyone. I'm Priyanka Gasliwal, solutions architect here at Databricks. And let me start with this is not a product talk. This is a practitioner story. Best narrated by Panasonic team themselves, who I worked very closely with over the last last year here and very pleased to share
[00:57] the stage today with Brendan, who's going to walk us through Panasonic journey. Over to you, Brendan. Thanks, Priyanka. Good afternoon, everyone. Thanks for joining. Uh my name is Brendan Byrne and I'm a data engineer with Panasonic of North America.
[01:13] And today I'm going to walk you through our journey transitioning from a set of legacy highly fragmented data systems to a modern unified platform built on Databricks. So, this journey wasn't just about adopting a new tool. It was about fundamentally rethinking how we ingest, govern, and deliver data across a large
[01:31] global organization. Along the way, we've unlocked significant performance gains, cost savings, and set the foundation for future AI-driven innovation. So, let me start by outlining what we'll cover today. The first half of this presentation will focus on Panasonic's
[01:46] experience. I'll walk through the challenges we faced with our legacy architecture, how we approached our Databricks implementation specifically using Lake Flow connectors, what our new architecture looks like today, the business value we've been able to unlock, the key lessons learned, as well as our
[02:02] future road map. After that, Preeti will come back and take over and provide the Databricks perspective. She'll cover common customer challenges she sees across industries, a deeper look at LakeFlow Connect and how it helps solve those challenges, the connector ecosystem and their future
[02:17] road map, a short Workday demo, and finally some key takeaways. But before we dive into the technical details, I want to take a moment to provide some context about Panasonic. So, Panasonic is a multinational electronics company founded in 1918 in
[02:32] Osaka, Japan, and over the years it has evolved into a highly diversified global enterprise. I work within Panasonic of North America, or PNA, specifically under Pexna IT. We operate as a shared services organization, meaning we provide centralized IT
[02:47] capabilities, including data engineering, to a wide range of business units. So, you'll see acronyms throughout this presentation, so here's a quick overview for each of them. PNA, that's our headquarters. PCEC, that's our consumer electronics division, most notably known for their Lumix cameras.
[03:02] Pesna is our residential and commercial HVAC, air quality, and sustainable building tech division. PAC is our aviation tech with our airline entertainment systems. Pexna is our EV battery production facility. We work closely with Tesla to create their EV batteries, and we work with them at
[03:19] their gigafactory out in Nevada. Pexa is our energy sales division. Pizza is our industrial systems division, producing electrical components for products like the iPhone and Roomba vacuum. And then Picona, which is the enterprise tech and connected solutions. We're actually using those products
[03:35] today. We noticed that these projectors are actually Panasonic, which is a nice surprise. So, to understand where we are today, we need to look at where we started. Our legacy architecture had several major challenges. First, we had data
[03:52] silos across both storage and ingestion pipelines. Different systems operated independently with little standardization. Second, we relied heavily on expensive third-party connectors, which limited flexibility and increased operational costs. Third,
[04:07] data extraction itself was a major bottleneck, especially for systems like SAP. For example, in SAP alone, we were running around 200 daily full table refreshes. These jobs were were time-consuming, resource-intensive, and prone to failure. Altogether, this created an environment
[04:24] that was difficult to scale, expensive to maintain, and slow to deliver value. But, given these challenges, it became clear that we needed a fundamental shift, a pivot away from this rigid siloed stack towards a more modern, flexible architecture. That pivot led us to adopt the
[04:39] Databricks Data and AI platform as the core of our new strategy. Our vision went beyond just improving pipelines, though. We wanted this platform to serve as the single authoritative data backbone that could serve all of our business units. This meant creating a platform where
[04:55] data is centralized, access is governed, and insights can be generated consistently across the organization. So, to guide this transformation, we established three core goals. First, we wanted to leverage data for measurable business impact. This wasn't
[05:11] just about modernization. We needed to clearly justify the investment with tangible results. Second, we aimed to streamline data ingestion and create a unified foundation. That meant reducing duplication, simplifying pipelines, and standardizing ingestion patterns.
[05:27] Third, we needed strong data governance, not only for compliance and security, but also to enable future AI and analytics use cases in a safe and scalable way. So, let's take a look at that first goal. So, to deliver impact, let's focus on a
[05:43] few high-priority data sources. In SAP S/4HANA, we're dealing with large data sets, long extraction times, and frequent job failures. In Workday, we're dependent on expensive third-party connectors, and Workday contains highly sensitive HR and payroll data, so data governance is key.
[06:01] SharePoint, where it's a little different than the other two, there are no major ingestion issues, but we found a huge opportunity in processing unstructured data. These systems gave us the biggest opportunity for improvement.
[06:16] Now, let's look take a look at how we streamline this in more detail. So, with SAP S/4HANA, our legacy system struggled with large SAP tables. Single table export run times uh frequently failed uh run times exceeded 2 hours, and they frequently failed. And these failures required manual effort and
[06:32] repartitioning to repair. If you're familiar with with SAP, one of the main tables that we had trouble with was the table ACDOCA, which is the core um line item table. This grows tremendously fast, and it's very difficult to manage. So,
[06:48] in our legacy system, the CDC option that we had was inconsistent. We would get dropped rows and missed updates. So, we needed a solution that way the business isn't struggling, and we could continue to engineer on uh and on more important items. So, the solution to this
[07:04] came in two parts. So, with a combination of upgrading to SAP Data Sphere and then using Databricks Autoloader, Lake Flow declarative pipelines, Auto CDC, and Databricks jobs, we were enabled to uh we enabled to shift from full extracts to incremental loads, reliably reducing
[07:20] execution time from 2 hours to just 2 minutes. This not only improved efficiency, but also freed up engineering time. So, working with SAP, it was one of our biggest challenges, and ultimately became one of our biggest wins.
[07:36] With Workday, our biggest challenges were cost and governance. Previously, we relied on third-party connectors, which forced us to limit ingestion frequency to control costs. To solve these issues, we replaced the third-party connectors with native Databricks Lake Flow connectors, which allowed more frequent and cost-effective
[07:52] ingestion. At the same time, we used Unity Catalog to properly govern sensitive HR and payroll data. The result of this was better data accessibility, stronger governance, and savings of $4,000 per month on just this one connector alone.
[08:13] SharePoint was a different kind of challenge. The issue wasn't about getting the data, it was using it. So, we had thousands of unstructured documents like contracts and master service agreements that required manual processing. We solved this using Databricks serverless compute combined with AI query capabilities.
[08:29] This allowed us to automatically process and extract insights from these documents. What would have taken 2 weeks of manual effort was completed in about 3 hours, fully automated. So, this slide shows a high-level view of our updated architecture. SAP data flows into the Databricks using
[08:47] Autoloader and Auto CDC. Workday, Salesforce, and ServiceNow use Lake Flow connectors. SharePoint and SFTP data use Autoloader with AI processing. And all the data lands in a medallion architecture with Delta Lake, and it is governed through Unity
[09:02] Catalog. Finally, let's take a look at governance. So, Unity Catalog is a cornerstone of our architecture. It provides centralized governance, auditing, and monitoring, fine-grained
[09:17] access control, and it's and one of its most powerful features in terms of optimization is row-level security and zero with zero copy architecture. Instead of duplicating data for each business unit, we store it once and control access dynamically.
[09:33] This is especially important for AI use cases, ensuring that users only see relevant data and avoid cross-contamination. Another major improvement was replacing SFTP based data sharing with Delta Sharing. So, previously cross-region sharing like
[09:50] between North America and Japan involved copy pipelines with tight scheduling requirements, difficult fixed with files to deal with, and strict formatting challenges. But, with Delta Sharing, data is shared directly across catalogs between regions. So, there are no
[10:05] intermediate steps, which means faster and more reliable sharing. This significantly reduced operational overhead. So, looking at our previous architecture once more, we could see that we moved from fragmented systems, redundant pipelines, high operational overhead
[10:23] to a unified, governed lakehouse platform with centralized ingestion and a scalable architecture. Business users can now easily consume data through tools like Power BI and data Databricks Genie from a single source of truth.
[10:41] So, the impact of this transformation has been substantial. SAP latency reduced from 5 to 6 hours to about 15 minutes considering all 200 tables that we were previously loading. SharePoint processing reduced from 2 weeks to 3 hours. And overall costs were reduced through platform consolidation. We've
[10:57] significantly improved both performance and the total cost of ownership. So, with any migration like this, there're going to be lessons learned along the way. And one of the first realizations we had was that solving SAP, even though it was our biggest pain point, was not enough on its own.
[11:15] If you think back to earlier, SAP S/4HANA was where we saw some of the most dramatic improvements. But, what we quickly learned is that if we only optimized SAP, we would still be left with expensive work Workday ingestion, manual processes in SharePoint, and disconnected ingestion pipelines across
[11:31] systems. The real value came when we stepped back and built a unified ingestion strategy across all enterprise systems. That's what ultimately enabled us to create a consistent scalable data foundation rather than just solving one isolated problem.
[11:46] Another key lesson was around how around how we implemented this platform. Initially, there's always a temptation to do a full Big Bang migration where we move everything at once and be done with it, but in practice, that approach introduces a lot of risk.
[12:02] It's harder to debug issues. It's difficult to maintain stability during this transition. Instead, we adopted a phased rollout strategy. What that meant for us was first establishing the Databricks foundation, our lakehouse or medallion architecture, and then Unity Catalog, and then
[12:17] gradually onboarding systems one at a time. Like SAP first, then Workday, and then SharePoint, and others. This allowed us to validate patterns early, reuse ingestion frameworks like Lake Flow connectors and autoloader, maintain governance consistently uh consistently from day one,
[12:34] and it also meant that the business could continue operating without disruption while we modernized behind the scenes. Another important takeaway was that stability is just as critical as performance improvements. Yes, we achieved major performance gains, but
[12:50] from a business perspective, what really mattered was that pipelines were running on time every time. We eliminated failures that required manual intervention and reduced operational firefighting. So, with the new architecture, pipelines became predictable and reliable. Engineering
[13:06] time shifted from maintenance to innovation. So, the real win wasn't just speed, it was consistency and trust in the data platform. And the final lesson, and arguably the most forward-looking, is that strong data governance isn't just a requirement, it's an enabler.
[13:24] Earlier, we talked about Unity Catalog and how it provides centralized access control, row-level security, and zero zero copy data sharing. This became especially important for two areas. First was business intelligence. Business units can access their own data without duplication now.
[13:40] Data is consistently defined and trusted, and tools like Power BI and Databricks Genie can operate off a single source of truth. Second, AI and advanced analytics. AI models depend on clean, relevant, and secure data. Without proper governance, you risk
[13:56] exposing incorrect or irrelevant data. And with Unity Catalog, we can enforce guardrails so each business unit only sees their own data, AI outputs are accurate and contextually relevant. So, instead of governance being a bottleneck, it actually became a key
[14:11] enabler for scaling both BI and AI use cases across the organization. So, when you put all of this together, what we really learned is that this transformation was about building a unified data strategy, implementing it in a scalable phased way, ensuring reliability and trust, and setting the
[14:28] foundation for future innovation with Databricks and AI. And speaking of future innovation, we got to look at what's next. So, looking ahead, our goal is to build a fully self-service data platform. We want business users to log into
[14:43] Databricks, query their data directly, and use AI tools to generate insights. If we continue along this path, we believe we could become the North Star for Panasonic to expand this platform globally beyond North America and become a global leader in modern data platforms. Now, with that,
[15:00] back over to Priyanka. Thank you, Brendan. What a journey.
[15:15] You heard firsthand from Panasonic how they went from data silos, legacy architecture challenges, pipeline and scale challenges. To a platform that lets you visualize, analyze, forecast, simply unlock your data.
[15:31] With the scale when you need it, with the ease of use for all personas of the organization. Let's ground us in the challenges that we heard from Brendan. Starting with the source volume. We're talking about hundreds of SAP critical
[15:47] data data tables here. We're talking about hundreds of millions of transactions flowing through every day. This is not a startup data problem. This is an enterprise-wide mission-critical data that's flowing, that's powering your manufacturing,
[16:03] finance, supply chain, audit. Here we are talking about all of this dealing with legacy CDC limitations, having to forcefully, brutally do full day refreshes because we can't figure out
[16:20] the surgical changes. We're talking about massive partitions causing pipeline collapses not once, not twice, 10 times a year that is roughly once a month where your data platform goes down, data analytics go dark.
[16:35] And for these outages to then be taken care of, full hours, full days of IT troubleshooting. Your data engineers, your infrastructure teams pulled in to fix these broken ingestion pipelines. And this isn't unique to Panasonic.
[16:57] We've all seen this. We've all done this on our Monday mornings. Ingestion is too complex but too important. Your custom scripts, manual orchestration, custom connector winders stitched together literally with prayers and cron jobs. It's too brittle. One field changes
[17:14] upstream, your pipeline breaks, no one knows until your dashboard goes empty. Here, we're talking about, you know, the whole governance is fragmented, multiple data copies. In your timetables, teaching zone,
[17:30] landing zone, taking care of, you know, drift risk, another line on your cloud bill. By the time data reaches your data analyst, it has passed five different system. No one knows where data originated. Was a PII masked?
[17:46] Is it the data copy that you want to work with? And in the meanwhile, your best data practitioners spending 60 to 70% of their time doing plumbing, retries, monitoring, orchestration, and not analytics, not AI.
[18:05] That's the problem Panasonic set out to solve. And Lakehouse Connect is how they did it. Databricks Lakehouse Connect is a native ingestion service, not a bolt-on, built-in. And three things made it work for Panasonic. First, it's managed, governed
[18:21] end-to-end, powered by serverless compute. Your data lands directly in cloud object storage, one platform, one security model, one lineage graph. It's easy. When you're talking about SAP, your data
[18:36] lands as soon as it lands, auto loader picks it up, infer schema, ingest in the Delta tables. It's completely simple. When we're talking about Workday, it's a point-and-click connector setup,
[18:53] replacing your expensive third-party connector, cutting cost in half. Third, we're talking about unified. Your ingestion and transformation live in the exact same UI, same alerting, same monitoring, same observability. As
[19:09] soon as your data lands, Unity Catalog, governs your data. No manual orchestration needed. Manual orchestration was problematic, hence Lake Flow Connect eliminated it.
[19:28] Let's look at a little bit under the hood here. When we are talking about serverless compute, smart auto scaling, what do we mean? When you're doing heavy SAP backfill, compute scales up. When you're doing a regular Workday incremental sync, compute scales down.
[19:45] You're not burning money for idle compute. System understands the exact incremental changes for the data source. You're leveraging auto CDC, cursor-based increments, or log-based changes.
[20:01] And the end-to-end solution is managed for you. Taking care of your automated failure recovery. System picks up where it left off. Such that your teams can actually focus
[20:17] on AI AI innovation and not figuring out doing troubleshooting and the maintenance work. Let's take a look at different types of connectors here. Starting with the SaaS connectors. Now, we have been talking about Workday,
[20:34] Salesforce, ServiceNow. Purpose-built for each API. Giving you pagination, rate rate limiting, schema inference, everything all handled for you, served by power uh served by serverless.
[20:50] Database connectors. Using log-based changes for your operation systems like SQL Server, PostgreSQL. For any of the sources where you can't enable CDC, use query-based connectors.
[21:08] We are using under the whole Lakehouse Federation such that you can directly connect to systems like Oracle, your Snowflake, BigQuery, Redshift, and lastly, community connectors. Custom connectors that you can write code using
[21:23] the templates that we provide for your niche internal data systems. With these four patterns, customers can virtually connect to any data source. And now this connector ecosystem is growing quarter by quarter.
[21:40] We're talking about hundreds of connectors as we heard in the keynotes today. Hundreds of connectors available today, your Workday, Salesforce, ServiceNow, different message buses, file servers available.
[21:55] And more coming soon. We're going to take a look at the Workday connector in action in a little bit here. Okay, so now we have all of this data flowing in getting adjusted from these different sources.
[22:12] We're talking about again SAP, Salesforce, ServiceNow. All of that landing in Delta tables. The next question organizations had is how am I going to share this across my organization? With my third parties without creating
[22:29] copies. That's the key part here, without creating copies. Panasonic, like working with so many different business units, they wanted to share data across different business units. Sometimes within same cloud,
[22:45] within same region, sometimes across cloud, across regions. The previous answer was let's create different storage accounts. We have the isolation per business unit, but then ended up having multiple data copies.
[23:00] With Databricks now in Unity Catalog, all you do is define the access control. And you get to share the data across your workspaces. One data copy, single source of truth with guardrails.
[23:18] If you want to share across cloud, across region, as Brandon mentioned earlier, Panasonic is using data sharing between the Panasonic North America and Panasonic Japan. What if I have data that I have not ingested yet in Databricks?
[23:36] We can use Lakehouse Federation for it to use data from the federated data sources where they are. Again, no copy. So, we have been talking about ingestion so far. Panasonic is more than an ingestion
[23:51] story. Let me zoom out. Let me go to the next slide here and show you the complete million-foot picture here. Enterprises work with dozen of different data sources at least.
[24:07] Starting from the left here, you know, organizations work with different formats, different APIs, different change patterns of these different data sources. And all of these data sources then with Lake Flow Connect,
[24:23] getting the ingestion done, having all streamlined landing in Delta tables, Iceberg tables in the open format in your cloud object storage. You own it. Full stop.
[24:39] It's not locked in into a proprietary format. It's not sitting in someone else's compute layer. That's a decision Panasonic made and that's why they can do that everything to the right of it where basically starting with the unified data and AI
[24:55] governance to be able to do auditing, lineage, semantics and everything that we heard today morning with the keynotes like different features that we we can do on top of once you have that
[25:10] unified core platform. Let me double-click when we are talking about work data, we were talking about sensitive data. To be able to do column masking, row-level security across BUs such that you're not again making a data copy,
[25:26] you're not complicating your data organization. You're just defining a policy on the single data copy. Here, let me also quickly emphasize Unity AI Gateway,
[25:43] which is a must today in today's data and AI landscape such that you can govern every single AI call irrespective of where that's hosted on Databricks, on Azure AI, OpenAI, or tapping into any third-party endpoint,
[25:59] you can register that within Unity AI Gateway and get the centralized controls of rate limiting, guardrails, everything. And then to the right of it, consume it anywhere. Your data analysts are sitting using Databricks dashboards, great.
[26:15] Your executives using Power BI, your finance team is using Tableau. Your data scientists are putting together notebooks. Our AI agents leveraging the Delta tables. Or your business users on their mobiles
[26:30] talking to Genie in natural language, not in SQL. Your end users decide the tools. Platform doesn't. Your end users, where they are, what's their comfort level,
[26:50] they decide the tool that they want to use for their purpose. That's a 360° view across BU's real-time single source of truth. And because it's sitting on open standards, open APIs, open formats, it's
[27:07] AI-ready today. So, when the landscape changes, and it will given, you know, how things are changing, Panasonic doesn't re-platform, they don't migrate, they just adapt because the foundation is right, foundation is open.
[27:24] That's the story we are talking about, not just automating ingestion. That's the end-to-end data and AI platform that's ready for whatever comes next. Here, let me pause here on the slide for a little bit. Um I want to emphasize for
[27:39] all of you to get further details on the Panasonic case study using this QR code.
[28:22] Oh, I uh I'll I'll come back to the slide uh towards the end and such that you get everyone can access. I I think on that particular screen, it's not completely visible, it seems like.
[28:40] So, we have seen the architecture, heard the numbers. Now, it's a demo time. We are going to start with the business value. We're going to basically see how a Panasonic chief HR officer
[28:55] can use our dashboard and Genie, can really use the Workday reports information. And we're going to peel back the layers of governance, ETL, and connector setup here.
[29:16] Here I'm logged in into our Databricks workspace at a Panasonic HR analytics dashboard. Of course, this is synthetic data. I I can't possibly have the Panasonic HR dashboard up here. Starting up top, when an HR officer comes in here, looks at this dashboard,
[29:35] having the top-level KPIs with the head count diversity, performance, all highlighted. We come one layer down here. It's broken down by a region, by tenant, and then further down just be able to see the trends and forecast and the attrition by
[29:51] quarter. Now, towards this bottom right, you can see the spike jumps out. This could take weeks on spreadsheets.
[30:07] And at the very bottom here, we see the diversity by region. All of this is powered by one govern governor of Workday. Workday data. We can marry that with different other sources. Right now, it's just Workday, but you know, essentially you can it's it's basically combination of
[30:22] different data sources and the answers that you want from the data. Dashboard is for the questions that you planned for. What about the questions that you didn't? For that, we have Genie.
[30:39] Panasonic is using Genie feature for their codes for the pricing terms and such. And here's the same idea for the HR. You can use that inbuilt within dashboards or you can also go within the Genie spaces here. So, let me go in here
[30:55] and then go to the Panasonic HR people analytics one where this is kind of a focused Genie space for this data. I can or rather HR officer can come in and ask what was our average headcount for
[31:12] Q1 of 2026. Or show me the attrition trends by manager hierarchy. They're not raising a ticket. They're not writing SQL. They're writing in plain simple language the question that they want to ask. I ran this attrition trends by manager
[31:28] hierarchy just for the sake of time here and um let me go to this next tab to show you. This is what Genie replies back with summary up top giving you the tabular
[31:44] results of the information that we asked and then the visualization in the bottom here. Now, the next question comes is how do I know from the accuracy perspective? I can come in here. I can look at the
[31:59] code. The SQL that Genie used to put this together. Also, I can look at the complete analysis that was done. Like complete thought process that Genie took to put together that information for you.
[32:16] But hold on a second. This is all the work that data that we are talking about it's sensitive data. What's happening with my governance? For that, let me go to the catalog here.
[32:32] So, I just went into the Unity catalog and I have a silver schema with the Panasonic HR data and within there I'm going to go into the active roster table where I have employee personal information here and I'm going to touch
[32:47] on like there there's a lot we can have we have so many sessions talking about the Unity catalog end to end. Here I'm just going to touch on some of the sensitive data policies where we have SSN here. So
[33:06] we can do column masking from the SSN perspective. Let me go to the column master defined here. And as you can see we're defining a policy to say HR admins can look at the whole value but for the rest it's always going to be masked and this
[33:22] is just policy that you configure at the catalog level at the schema level at the table level depending on where you want to configure and configure tags appropriately or have the system do auto classification but essentially the idea is you define
[33:38] it once and apply it everywhere where you are if you're accessing data on dashboards, Genie reports you're doing data sharing no exceptions the policy will be applied. Similarly
[33:54] let me go to the tenant row filter here where we're basically saying we have two business units at least in this policy I'm I'm configuring PNA and PECNA business units for Panasonic
[34:09] and we want to define row filter for those two when PNA analyst comes and logs in they can see PNA records when PECNA analyst comes in they see PECNA records. For HR admin they can see the whole view. But it's as easy as this and it
[34:27] simplifies Panasonic's HR data organization so much in not having to duplicate the data. So, that's a little bit about the policies when it comes to the sensitive data. Let's come to the foundation. How did we
[34:43] get the data in Databricks? I'm going to go into data injection here and then these are all the Lake Flow Connect data sources that you can see. Point and
[34:59] click functionality, you can use these, put that together, have the automated pipelines generated for you under the hood leveraging the CDC for each one of those. I'm going to take an example of the Workday reports here.
[35:19] And let's go ahead and create a connection to the Workday report. We'll basically create the connection, put the credentials, and I already have a connection created, so let me go to that. And um we basically select this, give it name to the pipeline. Let's see
[35:36] if I give Panasonic data pipeline, and then we basically define the event log location, like all the audit logs, everything um
[35:52] data quality checks, pipeline progress, errors, everything will go into this location. So, let's see. You're basically using Workday here, and we can go ahead and create the pipeline.
[36:12] Let's give it a minute here, and then what we'll do is the next step is to put the report URL that we want to ingest. Once we have given that the report URL, we'll basically define the destination within Unity Catalog. So, I just entered
[36:28] the employee review ratings report here. I can come in and I can decide the primary or given the primary key for that data. Do I want the history tracking SCD type two configured here or not? I can turn on and off depending on what I'm
[36:46] looking for here. And the next, decide the destination for it. Once I've configured where I want the data to land, the next step is to configure schedules and notifications. How often we want to
[37:02] run this pipeline? Every 24 hours, 12 hours, who to notify if anything failed, when there is success. You can just configure this all of these details
[37:17] in matter of these five steps, six steps. And I can go ahead and save it. The end result is the automated generation of this Spark declarative pipeline under the hood for you, showing you three different
[37:34] work data ports getting ingested, landing the data in the streaming table, and you have complete transparency. It's automated, that doesn't mean it's a black box for you. You have complete transparency as to when it ran, exactly what it ran, the duration, bytes read, written, complete
[37:50] details. All right. So, we have we kind of walked backwards. We start We started with the business value. We saw the governance. We saw the connector setup. Let me go back to my slides here.
[38:15] This is the whole flow we just looked at. Ingested work data ports. We use Lake Flow connector. Automated pipeline was put together, unified data then served on your analytics.
[38:36] Panasonic is such a massive enterprise proof point. Wasn't a POC. It's done at production scale. We're talking about multi-connector depth. Again, I know I've repeated multiple times, but we're talking about Salesforce, ServiceNow, Workday, SFTP, SAP.
[38:52] You name it and Panasonic is ingesting that data across functional scale. For your supply chain, manufacturing, audit, finance.
[39:10] Migrated with ease with our open wide third-party ecosystem with open APIs and of course grounded in our unified data and AI governance with the open format that's making the system AI-ready, future-ready.
[39:26] That's the end-to-end Panasonic story for you to take away. And on the right here I have some of the related day sessions that might interest you. In the same hall we will have another deep dive done on the Lake Flow Connect.
[39:42] Another one, SharePoint. I know Brendan kind of mentioned that data source there and we were not able to cover in depth because we touched more day, but there is a separate session happening on SharePoint. And then of course the agent-ready data governance one.
[40:02] Some of the Lake Flow Connect resources in the bottom there. Also, Brendan, myself, and the larger Panasonic team are here after the session for any questions that you might have. Thank you so very much for being here.
[40:18] I really hope you found this narrative actionable to experience this for yourself. To go from hours to minutes to seconds and milliseconds we heard today in the keynotes, right?
[40:40] to provide real-time updates to your end users on their fingertips. Write your own Databricks stories in coming months. Thank you so very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.