Databricks at Scale: How CDAO Built an 18,000-User Multi-Tenant Platform

Summary

CDAO, the Chief Digital and AI Office of the U.S. Department of Defense, serves 18,000 Databricks users accessing 98 petabytes of data from 700 authoritative sources, deployed across unclassified, classified, and top-secret environments.
In 2025, CDAO migrated from PVC to Databricks SaaS E2, moving 10,000 clusters and 8,500 jobs and rearchitecting to a multi-tenant deployment with separate infrastructure for the Department of Navy and Air Force.
Since enabling Genie in February, CDAO has generated 70,000 natural language queries across 1,000 Genie spaces, while Unity Catalog with CUI controls provides fine-grained governance for sensitive government data.

Databricks at Scale: How CDAO Built an 18,000-User Multi-Tenant Platform

The Chief Digital and AI Office (CDAO) operates one of the world's largest enterprise data platforms, serving 18,000 Databricks users across 98 petabytes of data spanning unclassified, classified, and top-secret environments. Their platform provides secure, scalable, interoperable data infrastructure for the U.S. Department of Defense.
Watch how CDAO migrated from PVC to Databricks SaaS, rearchitected to multi-tenant deployment for Navy and Air Force, and implemented Unity Catalog with CUI controls for fine-grained governance. Discover how Genie enables 70,000 natural language queries, serverless compute modernizes access strategies, and foundational components including infrastructure-as-code and observability drive real mission impact across supply chain, audit, and operational planning.
🤝

Chapters

FAQs

What is CDAO and how do they use Databricks?

CDAO is the Chief Digital and AI Office for the U.S. Department of Defense, responsible for deploying secure and scalable data infrastructure to accelerate decision advantage. They serve 18,000 Databricks users accessing data from 700 authoritative sources at 98 petabytes of storage across unclassified, classified, and top-secret environments.

How did CDAO migrate from PVC to Databricks SaaS?

In 2025, CDAO migrated from PVC to Databricks SaaS E2, migrating 10,000 clusters and 8,500 jobs — an undertaking completed even during a government shutdown. They also rearchitected from a single AWS account and workspace to a multi-tenant architecture with separate infrastructure for the Department of Navy and Air Force.

How is CDAO using Genie across the Department of Defense?

Since enabling Genie in February, CDAO has created 1,000 Genie spaces and generated 70,000 natural language queries, reducing the time users need to access and understand their data. Genie enables the distributed user base across combatant commands to query data without requiring technical expertise.

What mission-critical use cases does CDAO's Databricks platform support?

CDAO's platform supports supply chain management, audit, and operational planning across a supply chain with three times more suppliers than Walmart and more ground vehicles than FedEx. Common, trustworthy, and accessible data enables decision-making for a user base of 85,000 active tool users distributed globally across the Department of Defense.

Full transcript

[00:09] Hi, good afternoon everyone. My name is Aaron Mills and I'm here representing the Chief Digital and AI Office or CDAO. In a rapidly modernizing world, our mandate is to continue to stay at the forefront to deploy secure and scalable infrastructure and to leverage data to
[00:27] accelerate decision advantage. Delivering solutions at department scale introduces a bit of complexity. Understandably, our users are distributed globally located across the combatant commands.
[00:42] Our mission set is vast. We manage a supply chain with three times as many suppliers as Walmart and own more ground vehicles than FedEx. We employ more people than the entire population of
[00:58] Philadelphia. All love to Philly. I hear they're getting a new rail project. Instead of building disperate solutions to manage these disperate domains across supply chain, logistics, and people, we've built an enterprise data platform
[01:14] and a top that enterprise data products. We now rely on common, trustworthy, accessible data. We serve 85,000 active users across tools and 18,000 datab bricks users
[01:30] accessing data from 700 authoritative data sources at 98 pabytes of total storage. We've deployed data pricks across four impact levels and operate three production environments at uncclass,
[01:45] classified, and ts domains. We've unlocked the power of our data by serving as a core data layer to the department. Everything we build must be interoperable. Pri prioritizing open
[02:00] formats and integrations with other systems. We are setting design patterns that allow our distributed user base to ingest, govern, and share their data rapidly and securely. Evolution is a common thread across
[02:16] stories in this room. As technologies become available, we adopt. In 2020, we at CDAO began our journey with data bricks. And in 2025, we migrated from PVC to data bicks SAS offering E2.
[02:33] Migrating 10,00 clusters and 8,500 jobs is not for the faint of heart, especially during a government shutdown. Adding on a layer of complexity, it was also time for a platform rearchitecture. We had grown to a size where we could no
[02:50] longer operate from a single AWS account and a single data bricks workspace. In eight months, we migrated to a multi-tenant architecture, deploying separate infrastructure for our largest communities, Department of Navy and Air Force A4. Today, our customers are using
[03:08] natural language to talk through to their data via Genie. Since we enabled Genie in February, we already have 1,000 Genie spaces and 70,000 Genie queries, reducing the time required to derive
[03:23] actionable insights from data. Today in summer 2026, I hope to look back on this time and remember three cultural touchstones of equal importance. The Knicks winning the NBA finals. Go
[03:39] Knicks. Um the World Cup, of course, and our Unity catalog migration. Unity Catalog is helping us provision granular data access and distribute data stewardship. As you can imagine, we have
[03:56] many sensitive subtypes of data. We've applied the CUI framework for controlled and classified information as govern tags on assets. These tags are mandatory at asset creation. Data stewards approve CUI classifications and gate the release
[04:13] of control tables. To take on a little bit more during this migration, we are also enabling serverless compute. Serverless is required for our tenants to modify and modernize legacy access strategies. It also eases our path to
[04:30] future feature adoption as a lot of the capabilities we've been hearing about here require it as a prerequisite. Pre-Unity catalog, one of our use cases created 6,000 distinct views to
[04:45] customize which ERP data their different consumers were able to access. Now with column masking and row filtering, we maintain stringent data access policies, validate compliance through a policy engine, and minimize
[05:02] and reduce complexity. Through Unity catalog, we provide a unified entry point to consumers and reduce the amount of data copy when sharing with partner systems. We grant access directly to the managed tables
[05:18] and to their associated metadata. This means the CUI tags, limited dissemination controls and distribution statements are read alongside the data and access is governed appropriately in downstream systems. We've completed a successful pilot with Foundry reading
[05:35] Unity catalog data at lower latency, reading the tags and writing back to our core data layer. This governance layer provides maximum flexibility to distributed organizations. We at CDAO are able to
[05:51] focus on democratization and interoperability. We don't need to be opinionated on where our users are accessing data as we are able to securely share it. Four foundational components have enabled us to operate at the scale we do
[06:07] today. Mission spaces allow our largest consumers to deploy and operate their own instances of tools and services in a dedicated AWS account. Our core data layer is accessible across accounts.
[06:23] Infrastructure is code. As we modernized our platform architecture and our data storage strategy, we needed a declarative version controlled approach to infrastructure deployment. As an example, our S3 GitHubs pipeline
[06:39] enforces platformwide standards provides a complete audit trail when provisioning buckets. We built a suite of data movement services as common services all of our users are able to access. These are AWS
[06:55] native event driven. But in 5, our uncclass environment, we needed to modernize those to read from Unity catalog. We maintained backwards compatibility in our higher level environments on the high side where E2
[07:10] is not yet available. We are actively working with data bricks on that partnership to make E2 available in zipper and JWIX. Lastly, observability. We allow our c customers and consumers access to see usage and spend and do
[07:28] this through cost center tags and enterprise dashboards with drill down ability into high-cost workloads. I want to leave you with mission impact. We have built this data foundation. Now what problems are we using it to solve?
[07:46] Commonly understood data improves operational planning. We see this in common operating pictures developed across classes of supply. These applications illuminate supply chains and provide scenario planning capabilities.
[08:01] While products built around parsed naval message traffic provide Navy leadership with real-time visibility into ship status, which then improves short and long-term planning. Trustworthy data promotes a clean audit.
[08:17] Comproller is establishing authoritative lineage, resolving gaps in transaction level visibility and improving reconciliations. These applications save time. Headquarters Marine Corps streamlined a
[08:34] manual quarterly reporting requirement. In doing so, the Marine Corps recovered approximately 200 manpower hours from uniform service members each quarter. And most importantly, these products have real world impact. CDAO supports
[08:51] non-combatant evacuation operations in times of crisis alongside partner agencies. These are just a few of the examples of the breadth of problems we are solving today. Now that we have established and laid the foundation with a core data
[09:07] layer, I can't wait to see the problems we solve tomorrow.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.