Build Real-Time Manufacturing Dashboards with Databricks and CDC

Summary

Anduril's data engineering team delivered sub-minute latency manufacturing dashboards in just 8 weeks using Databricks on GovCloud, meeting strict federal requirements that ruled out most commercial SaaS tools.
The architecture moves raw CDC data through Spark structured streaming and Auto CDC flows to dimensional tables and continuous materialized views, producing live factory floor displays in near real time.
The team took a pragmatic approach that prioritized delivering business value over perfect technical elegance, enabling visibility into inventory levels and equipment effectiveness across the production line.

Build Real-Time Manufacturing Dashboards with Databricks and CDC

Anduril manufactures complex defense systems under strict federal constraints, requiring real-time visibility across manufacturing operations. In January, the company faced an urgent challenge: deliver sub-minute latency reporting to a new production line in just 8 weeks.
Learn how Anduril's data engineering team built a real-time manufacturing dashboard using Databricks, Spark structured streaming, and CDC capabilities in GovCloud. Discover the architecture that moved from raw CDC data to dimensional tables using Auto CDC flows and continuous materialized views, and the pragmatic approach that prioritized delivering business value over perfect technical elegance.
🤝

Chapters

FAQs

What challenge did Anduril face that led them to use Databricks?

Anduril needed to deliver sub-minute latency reporting to a new manufacturing production line in just 8 weeks. Existing SQL-based batch frameworks could never meet the one-minute latency requirement, and building a custom streaming stack was too time-consuming given their deadlines.

How does Anduril's real-time manufacturing dashboard architecture work?

The architecture processes raw CDC data through Spark structured streaming and Auto CDC flows to produce dimensional tables, which are then surfaced through continuous materialized views. Factory floor displays show live inventory levels, equipment effectiveness, and other operational metrics in near real time.

Why does Anduril deploy its data infrastructure on GovCloud?

Anduril is a defense technology company subject to strict federal requirements, which means most commercial SaaS tools are unavailable from day one. GovCloud compliance is mandatory for all infrastructure supporting their defense manufacturing operations.

What is Arsenal OS at Anduril?

Arsenal OS is Anduril's unified system connecting engineering, production, factory machines, and field feedback so information flows without waiting for humans to manually move it. It is the platform through which Anduril builds hardware at software speed.

Full transcript

[00:08] I want to begin by asking you to imagine that you work at a defense company that is rapidly scaling production on a range of technically complex systems. Alongside the actual engineering that needs to happen, you need to optimize the systems that comprise your industrial workflow, and the timelines to deliver outcomes are only getting
[00:24] shorter by the day. To accomplish this means connecting outputs from dozens of systems in countless shapes, and taking that data and making it useful. All while dealing with security controls and federal constraints. And then once you have that data,
[00:39] uh taking it to the floor and making it useful to all the folks that work on the floor. Welcome to a regular day at Anduril. My name is Cullen McCann, and I'm the head of Arsenal OS data at Anduril. There we go. Uh Anduril is a defense technology company whose mission is to transform
[00:56] national security uh with advanced technology, from autonomous submarines to fighter jets to mixed reality headsets. And Arsenal OS is the way that we build that hardware at software speed. One system connecting engineering, production, factory
[01:12] machines, and field feedback, so that nothing waits for humans to move information. In January of this year, my team was tasked with delivering sub-minute latency reporting to a new manufacturing line producing hardware important to national security. We'd never done it
[01:28] before, and the need was urgent. Every week without visibility made it harder to meet our deadlines. This is the story of how we accomplished this in 8 weeks with Databricks. So, the fun thing about building data infrastructure at Anduril is probably
[01:44] familiar to most folks in the room, and that most of the standard choices don't work for us. Everything we use has to be deployed under GovCloud under strict federal requirements, which means that from day one most commercial SaaS tools are out. Our team had already standardized on
[01:59] SQL-based frameworks in the past, but they could never meet the requirements that we needed for this project because they were at batch. They'd never meet that one minute latency. We considered spinning up a custom streaming stack, but that would have meant building a parallel universe, and given the deadlines, we just didn't have
[02:15] the time. So, what we needed was simple for our stakeholders to describe. Live dashboards or displays on the factory floor showing the state of the manufacturing line in near real time. Inventory levels, equipment effectiveness, issue on the line, all
[02:30] updating within the minute. Simple to describe, not so simple to build in the environments that we operate in. So, this is when we started talking with Databricks. We kicked off the conversation with them in January and wasted no time getting started. With support from their team,
[02:47] we had a developer instance deployed on our infrastructure in just under 3 weeks. That alone was an accomplishment by both teams. Most vendors take months to clear to clear our security reviews, let alone weeks. From there, the architecture came
[03:02] together pretty quickly. One of our data engineers set up a change data capture or CDC connection to our manufacturing execution system, which enabled us to capture every change as it was happening. We streamed that raw change data into Databricks, used Spark
[03:17] structured streaming, broke it out into individual data sets, and within 3 weeks we had data flowing at the latency that we needed. But, the thing about CDC data is that it tells you what changed, not what the current state is.
[03:33] Uh we had a deep log of inserts, updates, and deletes, but that actually didn't matter to the folks on the floor. What they needed were actual dimensional tables, work orders, inventory counts, work center statuses, all enriched with other data sources. And to build what
[03:48] they needed, we had to join across dozens of relationships. It was these joins that gave us the most problem. Every stream we tried to set up that joined together all of the necessary dimensional data ended up breaking our SLA almost immediately, to say nothing of the technical complexity of managing
[04:04] all of those checkpoints. The business needed results and we didn't have time to experiment. At this point we were about 6 weeks in and it looked like we didn't have a clear path forward to delivering outcomes. And that's when we discovered Auto CDC flows. It's a capability within
[04:21] Databricks that takes raw CDC data, applies primary keys, and then automatically reconstructs the current state of the table continuously in real time. That was the unlock. Suddenly we had the current state of every table without needing to manage it ourselves,
[04:36] without needing to deal with those checkpoints I mentioned. Once we had these we were able to layer continuous materialized views on top of them to handle all of the joining across the different data sets that I mentioned running every 10 seconds. Uh, two more weeks of development, testing,
[04:52] and validation followed and then we had it. Event to dashboard under a minute. Eight weeks after kickoff the dashboard was live on the production floor. Operators are now able to glance up and see all of the context that they need.
[05:08] Uh, quality issues, overall performance, status of the line, and more. And this knowledge is critical in a very fast-moving environment. But as the foundation shifts underneath us in this fast-moving environment, we recognize that not every solution we are going to
[05:23] build is going to be the most technically elegant. Uh, there may be streaming purists out among this crowd who are looking at what I'm describing and recognizing that there are dozens of things we could have optimized or savings we left on the cutting room floor. And you're right, but more important
[05:40] than getting to the perfect technical solution is getting to the solution that works for the business fast. And that's what we've been doing and it works. We've been running this for 3 months at this point, and most importantly, it met the timelines that the business needed.
[05:56] Taking an approached approach rooted in rooted in practicality also gave us much-needed flexibility. We didn't have to spend weeks perfecting one pipeline before we even knew what the most important metric on the floor was going to be. We could bring in late-arriving data, handle any degree of
[06:11] normalization, and add new sources without needing to re-architect or spend weeks on each new data set. When you work in an environment as fast-moving as Anduril, where the product and the process are evolving constantly, you learn very quickly that assumptions are
[06:27] very expensive. Flexibility is what lets you keep pace. An old manager of mine was fond of the adage, "Make it work, make it right, make it fast." Getting real data in front of real operators on the floor was more valuable than building the perfect
[06:43] technical solution in isolation. You can't improve what people can't see. Now, everything I've described is only possible because of the Anduril team behind it. Uh Morris Lee, Thomas Hill, Juliana Altabeli, and Brian Krantz. They built this under real pressure, and they
[06:59] delivered. But it was also possible because of the Databricks platform. It had depth when we needed it. Every time it hit or it felt like we hit a dead end, there was another capability behind a door that ended up solving the problem that we were running into. And what did we learn
[07:15] from all of this effort? There's a world of difference between go run three queries and piece the data together, and glance up at the wall to know what's happening. 2 hours and 1 minute are not the same thing. And the approach we landed on is a scalable one. Other
[07:31] production lines across the company have seen the success that we've had and are breaking down our doors to get this rolled out. Literally, as I stand here, deployments are happening at other product lines, and we're just getting started with what we're planning to do with the Databricks platform.
[07:46] Uh we're going to keep moving forward with rapid iteration. The team has spent the last 2 months uh migrating all of our legacy pipelines over to Databricks so that we can begin to utilize capabilities like Genie and apps, all resting on top of fine-grained uh fine-grained access controls
[08:02] implemented with Unity Catalog. And as soon as the Databricks team makes Lakehouse available to GovCloud customers, we plan on throwing dozens of workloads against it. If there's one thing I want you to take away from this talk, it's this. A deep
[08:17] platform plus a motivated team can deliver on timelines that others would call unrealistic. Stability and flexibility let you serve the business, not just design a perfect architecture. And when you're building hardware that the warfighter depends on, serving the
[08:33] business isn't optional, it's the mission. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.