From Code to Insights: Leveraging Advanced Infrastructure and AI Capabilities

Summary

• Manufacturing analytics: This video shows how Insulet uses Databricks to reduce scrap part costs through advanced manufacturing analytics, demonstrating efficiency and cost savings.
• AI-assisted development: Databricks AI solutions assist with code debugging, query optimization, and error detection, providing actual solutions beyond basic suggestions.
• Real-time data integration: Using Databricks Lakeflow Connect enables seamless, real-time integration with Salesforce and processing of patient data for operational efficiency.

From Code to Insights: Leveraging Advanced Infrastructure and AI Capabilities

Talk by Shweta Shetty, Director, Insulet
In this talk, we will explore how AI and advanced infrastructure are transforming Insulet's development and operations. We'll highlight how our innovations have reduced scrap part costs through manufacturing analytics, showcasing efficiency and cost savings. On leveraging Databricks AI solutions and productivity, it not only identifies errors but also fixes code and assists in writing complex queries. This goes beyond suggestions, providing actual solutions. On the infrastructure side, integrating Spark with Databricks simplifies setup and reduces costs. Additionally Databricks Lakeflow Connect enables real-time updates and simplification without much coding as we integrate with Salesforce. We'll also discuss real-time processing of patient data, demonstrating how Databricks drives efficiency and productivity. Join us to learn how these innovations enhance efficiency, cost savings and performance.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How does manufacturing analytics reduce scrap costs at Insulet?

Databricks enables analysis of manufacturing processes and quality metrics, identifying inefficiencies and failure patterns that drive scrap. This intelligence guides process improvements and cost reductions.

What AI capabilities does Databricks provide for developers?

Databricks AI identifies code errors and suggests fixes, assists in writing complex SQL queries, and provides actionable solutions rather than suggestions—improving developer productivity.

How does Lakeflow Connect simplify Salesforce integration?

Lakeflow Connect enables real-time data updates from Salesforce into Databricks without extensive custom coding, simplifying data integration and reducing infrastructure complexity.

Why is Spark integration with Databricks valuable for Insulet?

Integrating Spark with Databricks simplifies distributed data processing setup and reduces operational costs while maintaining scalability for enterprise data workloads.

Full transcript

Hello everyone. Welcome to the session. Great first day. I hope everybody is soaking in all the new stuff and uh all the news you guys are hearing here. Um I'm super excited to be here. This is my first time. Um but uh happy that you guys are here for my session. Uh before we dive in, just a quick show of hands, how many of you know about Insulet or Omnipod 5? All right. Um just a brief introduction. Insulet was founded in 2000.
Um it's uh headquartered in Acton, Massachusetts. And um our mission is to simplify lives of people living with diabetes, whether it's type 1 or type 2. Really uh getting people out of those multiple daily injections. Um we are best known for our flagship product called Omnipod 5. It's the number one automated insulin delivery system in the market. And it's wearable, it's tubeless, and it's cloud connected. Born from a father's desire to improve his son's life, Insulet continues to innovate uh with compassion and purpose at its core, really.
And here I am. I'm Shweta Shetty, Director of Data Engineering at Insulet. Um I lead the data teams and my team builds and manages the data infrastructure. Uh really, which is fueling the product innovation and data insights within the company. All right. Um just to give you a sense of who we are and what we do, I just wanted to share a short video with you guys and to just show you a spirit of Insulet. Here we go. All right.
Uh every time I see such videos, it just makes me believe that I'm in the right place and doing the right things. You know, at the heart of everything we do, and really everything, even those endless meetings, is our rock-solid dedication to our customers. We also have strong values to deliver our product with quality, compliance, integrity, and really we all share the passion and commitment to fulfill our mission. We have been on a rocket ride success on our product Omnipod 5 since its launch.
Uh the biggest key differentiators for us is um the form factor and the automated insulin delivery system. And we want to reach a point where data is going to be our next competitive advantage. And as you know, the opportunities to use data in an effective and really impactful ways are endless. So, before we started in Databricks, um we were all things Azure. Our data lake was in Azure with Azure Data Factory, Azure Synapse, Azure SQL database for our metadata management.
Really, we were bunch of hopefuls in Azure with a dream. But um our Databricks journey began in 2019. Uh we were yeah, just trying to consolidate everything back into the Databricks ecosystem, so it's all in one um data platform. Little did we know that Databricks was also trying to simplify our lives, data data engineers' life, right? Um Come 2020, Omnipod 5 launched, and I uh really our customers were waiting for this product for months and years.
And as you know, with a with a successful launch of the product, there are more customers and more valuable data in our system. The partnership with Databricks was strategic here, and thankful to the Databricks team for us to manage all the data demands we had, and it paved the path to success and helped us better manage our data in real-time processing and as well as improving our operational efficiencies as you can see here.
So, where are we now? Um here's a really high-level uh architecture diagram showcasing our current landscape using Databricks in a nutshell. We have multiple sources which we are ingesting with Databricks Intelligence Platform and Databricks Unity Catalog. We are currently enabling our business users to analyze the data which they are authorized for. The initial movement to Databricks was to move as a compute platform and slowly we are trying to move it all our data all the pipelines from Azure Data Factory and slowly uh bringing it to Databricks workflows as Databricks came out with those uh for us and it was quite a plug-and-play really and the orchestration layer for starting to ingesting all these from the sources as mentioned was pretty straightforward.
And we have been privileged to work with Databricks team getting the private previews the features they want us to test maybe and then and for us to actually deliver for our customers was very critical. One such uh feature which I want to dig deeper into is a Lake Flow Connect. This has been a game-changer. Um it's efficient, it's incremental ingestion which drives our analytical ecosystem downstream. And in addition to this, we also were very successful in ingesting our complaints database which is a Track which is in Trackwise which is again a Salesforce connected app.
So, again very seamless uh really low-code low-maintenance which really helped the team focus not on the ingestion layer but on the downstream curated data sets and enhancements. Moving on to Databricks Catalog, I have no doubt that it will pave us the path to data governance. And with all the news uh and all the new features they're coming out out with, um I'm really looking forward to creating a company-wide catalog with centralized access and permissions.
It just gives a unified visibility of all the data for us. And uh as you know, we are a manufacturing company as well. We manufacture our pod. And for our near real-time manufacturing analytics, we currently connect to our on-prem SQL database um using Databricks. And then uh we use Data Factory today, but really uh right now we are leveraging data uh Databricks for the compute, for cleaning, for merging, and the downstream analytics.
But we are actively transitioning to Lake Flow Connect uh with the SQL Server and CDC uh enabled. And I I bet this shift is going to dramatically simplify our architecture and make our data pipelines really far more robust and scalable. Again, hey, we are not stopping here. We are still exploring for our real-time deep device data. As we know, our pod and our uh applications, the Android and the iOS, they're all cloud connected.
Uh we are exploring real-time device data ingestion from the device data, which will allow us to respond faster to our customers' needs, right? To drive innovation at the edge. And here's the big picture. We are looking into Databricks not just for our online analytical processing. We're also looking at online transactional processing with the Postgres news, which we are we're going to they're going to come out with, right?
This future of data platform is really bright. And I'm yeah, we are really looking to build it uh using all these new features. So, in the data maturity journey, after building the solid data ingestion foundation, we decided to take things up a notch. We entered the data mesh bandwagon because that's This is one of the success stories in our company, and if anybody wants to know about more about it, let me know. This was driven by the data domain, which had expertise on the Salesforce data, and we started with the Salesforce data getting augmented with some of our data vendors, third-party data vendors.
The key KPIs were really delivering high impactful provider engagements, sales effectiveness, and territory performances, right? And these were just not dashboards, but decision decision engines. So, one of the things which we did was, "Hey, let's set up best practice architecture around this." So, we followed the medallion architecture best practice. As you all know, the gold, silver, and bronze layers. Uh so, the curated data sets were were uh the curated insights.
And each domain had their own workspace, so ensuring they have autonomy and without any chaos. And yes, we had governance also enabled, so because with great data, you have great responsibilities. And we didn't stop there. Finally, we published it into these curated assets into the Unity catalog, which was fully governed and discoverable. So, it was like putting a bow on a really nice data gift, ready to be consumed and secure by design, right?
So, once we had the data mesh foundation in place, Uh next challenge was path to production. How do we deploy our jobs, workflows securely, reliably, and in a GXP compliant way? GXP is a good practices for a medical device company. We follow that. And our developers really work in an isolated environment where they can test, experiment, and iterate freely, right? We have version control with uh Git practices. We don't no longer have final Oh, finally files floating around.
Um again, I come from a very software engineering background. Really want to get the best best practices in the data engineering side as well, right? No-brainer. So, when a developer is ready, um we are ready to promote changes, they follow the structured Git flow. Feature branches, pull requests, code review, the whole nine yards. And again, we use Databricks access bundles uh to securely package and deploy those resources.
Again, under the hood, Terraform was taking care of the provisioning of the work the provisioning of Databricks workflows, jobs, configurations. So, we're not manually clicking anything in the UI at 2:00 a. m. in the night. You don't want to do that. So, every step of the process is validated because we are GXP compliant again, good practices from code to deployment. Ensuring traceability, reliability, auditability, and of course, peace of mind, right?
Next, uh I wanted to share one of the another features which we use with Databricks is one of the most exciting shifts, I think, uh in the data journey, at least my data journey, has been leveraging the Delta sharing as a data consumer from external data vendors. Especially when Databricks adds this functionality and makes it more available. So, in the past, I bet everybody knows, ingesting curated data set from partners meant really complicated ETL pipelines, often duplicating efforts and introducing delays, right?
Now, with Delta Share, we simply connect and consume. I mean, how simple it can be, right? No heavy lifting required. Why it's a game-changer? I think it's secure, it's compliant, and real-time, right? With no unnecessary data movement. We get access to high-quality data, curated data sets directly from our partners, ready to use for analytics and machine learning workflows. It does have a high business impact. This has drastically reduced our time to ship and time to insights.
Um and all the over engineering overhead, like you don't have to build this APIs or uh get in like ingest through APIs, you have it ready in the Delta Share. It really has opened up um doors to cross-organizational collaboration like never before, in my opinion. And I'm really looking forward to do more of this even as a consumer and a producer within the company. All right, but wait, let's do some reality check. As exciting as the innovation sounds, we need to keep an eye one eye on the prize and the other on the budget, right?
Um we really have to see what we are um what is the return of investment, right? ROI, like what what are we using and how are we using and what's the cost associated with it. We really embrace the FinOps approach. Uh Uh again, treating cloud costs not as an afterthought, but as a first class citizen within our architecture. Because let's be honest, no one wants to be surprised by that data pipeline costing you more than a Ferrari, right?
Um through our data maturity in our company, enabling cost observability was was the first first task, right? With the help of Databricks team again, I'm really thankful for this because we leveraged system stable to gaining deep visibility into the usage metrics and the cost drivers. This really gave us the ability to monitor, analyze, and really optimize our spends in near real time. All the dashboards are created, we we review it, and we figure out which one is a top one.
Even to the point where you can actually pinpoint which user is is is costing us, right? So, so our data ops team played a huge role with just one person in my team enabling resource tagging, right? We could trace the cost back to the teams, to the projects, and even to the individual users. This collaboration was made possible between the FinOps, the data ops, and the Databricks team all championed to this journey and helped us bring this to life, really.
And and everybody should do this in their data journey. Um because you will be asked at some point. Anyways, as we continue to innovate uh and grow, let's let's celebrate what truly uh drives us forward, right? The power of data and the great meaningful change. And thank you to the incredible team here in the Databricks to great partnership for this impactful journey. Yeah. Thank you.

Learn more about the Databricks Data and AI platform.