Governed Multi-Tenant Platform: Scaling Databricks to 3,000 Users

Summary

First American Financial runs a governed multi-tenant Databricks platform on Azure serving 3,000+ users across 24 teams, using a structural tenancy model rather than policy-based governance to prevent the governance collapse that typically occurs when enterprise platforms scale.
The platform treats all infrastructure as code, deploying changes atomically across three planes: Terraform for Unity Catalog governance, Alembic for schema management, and Databricks Asset Bundles for runtime resources, all orchestrated through GitHub Actions.
The structured approach reduced new team onboarding from months to days and achieved 100% audit traceability, while Lakeflow Connect consolidated vendor data ingestion and laid a foundation for future agentic automation and zero-ops pipelines.

Governed Multi-Tenant Platform: Scaling Databricks to 3,000 Users

Regulated financial services struggle to balance data democratization with governance at scale. First American Financial runs a governed multi-tenant Databricks platform serving 3,000+ users across 24 teams. The architecture separates interactive analytics from production data services, using clear tenancy boundaries to prevent the governance collapse typical at enterprise scale. The key: structure over policy.
Learn the implementation: a monorepo strategy with three deployment planes (Terraform for governance, Alembic for schema, Databricks Asset Bundles for runtime) orchestrated through GitHub Actions. Discover unified data ingestion that consolidates vendor sprawl using Lakeflow Connect for managed sources and a reusable Python framework for custom patterns. See measurable results: onboarding acceleration from months to days, 100 percent audit traceability, and a foundation for agentic automation and zero-ops pipelines.
🤝

Chapters

FAQs

Why does governance collapse when enterprise data platforms scale?

First American Financial observed that governance collapses when teams grant each other access ad hoc because it is the fastest path forward, and pipelines accumulate without clear ownership. Policy documents cannot keep pace with growth, so the only reliable solution is structural tenancy—boundaries built into the platform itself that hold even as teams and users are added.

What are the three deployment planes in First American's platform architecture?

The platform uses a monorepo deployed atomically through GitHub Actions across three planes: the governance plane manages Unity Catalog configuration using Terraform, the schema plane handles database schema evolution using Alembic, and the runtime plane deploys jobs, notebooks, and pipelines using Databricks Asset Bundles.

How did First American Financial accelerate new team onboarding?

Before the structured platform, onboarding a new team took months of manual setup and access negotiation. The tenancy model and code-based deployment pipelines reduced onboarding to days, since governance boundaries, schemas, and access patterns are codified and can be instantiated automatically for each new tenant.

How does Lakeflow Connect fit into First American's data ingestion strategy?

First American previously had significant vendor sprawl for data ingestion across 24 teams. Lakeflow Connect provides managed connectors for standard data sources, while a reusable Python framework handles custom ingestion patterns, consolidating the ingestion layer under unified governance without requiring each team to build bespoke pipelines.

Full transcript

[00:07] Morning everybody. I'm going to get started. Um the uh front matter that we've seen a couple times now, a forward-looking statement. Um you get a survey with each. What we're talking about today Um so I'll introduce myself. Uh my
[00:23] name's uh Brett Beavers. I'm senior engineering manager at First American Title, where I lead our strategic technology teams data engineering function. My team builds and runs the SD2 Lakehouse, a governed multi-tenant Databricks platform
[00:38] on Azure that serves data to several thousand users and a couple dozen teams across the company. In a regulated financial services environment, our job is to democratize data without ever losing our governance posture. So we put a lot of thought into how we
[00:54] operate our system while modernizing them at the same time. So let me start with the scale we're operating at. Today we run one Databricks platform that serves more than 3,000 users across 24 teams.
[01:09] That's analysts, data scientists, and business partners. Uh we're financial services company, so that data includes sensitive regulated information, and we answer to auditors about who can handle that information. The tension is every one of those 3,000
[01:24] users wants to move fast. They want to explore, build dashboards, ship pipelines. But the more people you let in, the faster you let them move, the harder it gets to keep governance intact. So the question is this. How do you scale Databricks to
[01:40] enterprise scale without governance collapsing? So we'll address the question in parts. First, I want to be specific about why governance collapses at scale. Uh because if you've run a platform like this, you probably have watched it happen.
[01:56] Second, the tenancy model. This is our answer to the problem. It's a structural boundary built into the platform itself that holds holds even as you add teams and users. It's a structure, not a policy or a wiki
[02:11] page. And third, we treat the entire platform as code. We deploy changes atomically across three planes, governance, schema, and runtime. Finally, my co-presenter, Sujith, will explain how all this is applied to real
[02:28] area practice, data ingestion. How the structure enabled us to move more quickly to consolidate ingestion with LakeFlow. So, here's the problem. Uh it almost always begins the same way. A few teams stand up workspaces, they
[02:44] start building. Pipelines accumulate, people grant each other access as they need it, because that's the fast path, and nobody wants to be the bottleneck. And it works for a while. The platform feels unencumbered and productive, and adoption grows.
[03:00] Then about 18 months in, say, the governance collapses. And it's not one dramatic failure, it's an accumulation. Permissions are inconsistent from one team to the next. Ownership is unclear. Nobody's quite sure who's responsible for what. You might need to make an urgent
[03:16] cross-cutting change, but you have a sprawl of ad hoc environments with different architectures. This progress is predictable because the same speed that drove adoption is exactly what erodes control.
[03:34] This erosion isn't just technical debt. It creates real operational problems. Some of our pipelines necessarily handle regulated personal information. There's a small number of people that require access to maintain those pipelines, and some of our users necessarily need broad read access to do their jobs.
[03:49] Those two populations cannot be confused. The risk is blast radius. Uh a misconfigured permission grant or a bad pipeline run should never take down your production product or expose data across the platform. Blast radius is exacerbated by privilege
[04:07] drift. In any system where humans grant permissions ad hoc, entropy wins. Nobody remembers why a particular AD group had right access to a particular schema. And all this complicates auditability. When an incident hits, you need to know
[04:22] who owns it, where it's allowed to run, and what it's allowed to do. Now, it's tempting Oh. Sorry, I lost my scale here. It's tempting to address these with policy, but policy doesn't scale. It doesn't survive organizational change.
[04:39] So, we made the choice to address it with structure. So, how do you actually stop the collapse of governance? When governance starts to fail, the instinct is to add process.
[04:55] You write a wiki page that explains how environments are supposed to be set up. You appoint reviewers to enforce it. You add a approval step, and that feels like progress. But process governance doesn't sail scale. Policy that lives in people's heads and in documents always lose to entropy. So,
[05:11] our strategy is to govern with structure instead. If you draw the boundaries correctly up front, the policy mostly enforces itself. That's our tenancy model and what it does for us. So, the whole model comes down to three primitives.
[05:27] Workspaces, tenant types, and catalog types. As I'll explain in the next slide, workspaces are the control plane. Tenants are the ownership boundaries, and catalogs locate data and resources.
[05:44] So, first, workspaces. A workspace is the Databricks control plane. The UI, the notebooks, the jobs, and all the resources they access. We deploy workspaces in groups of three. The interactive workspace is where humans operate. The non-prod workspace is where services
[06:01] get tested and invalidated. Service principals run jobs and UAT happens. Uh very limited human access. And the prod workspace is touchless. Production services execute under service principal identities and end
[06:17] users never log in. Uh second tenant types. A tenant is an ownership boundary uh an identity plus the resources the identity owns. An interactive tenant is owned by an AD group
[06:32] uh representing a team or business unit and has no environment life cycle. There's no prod version of a team's notebooks. A service tenant is owned by a service principal. It's deployable unit of software with full dev, non-prod, prod
[06:49] life cycle. And third catalog types. Service catalogs are private, owned by a service tenant, and they may contain encrypted NPI. Shared lakehouse catalogs are governed data products, broadly readable, they
[07:05] contain no NPI. And this last constraint is the most important thing on this slide. The shared catalogs are NPI free by construction. Once you commit to that, publishing to the shared catalog becomes a real security gate.
[07:26] So, let's make this concrete concrete. A business called Fusion wants to onboard to the platform. Their engineers want to ingest data from source systems and build medallion architectures with ETL. Their analysts want to build reports on the gold data and do ad hoc analysis on data they upload themselves. So, what does Fusion actually need?
[07:43] In our model, it's two things. A service tenant for engineers and an interactive tenant for the analysts. The common pushback is why two things? Why not just give everyone a service tenant? It handles every case
[08:00] but at the cost of forcing every team through CICD lifecycle that they don't need. A business analyst produce a building a dashboard does not have a dev non-prod prod story. They have a dashboard. Artifact promotion is critical to
[08:16] controlling risk to production systems, but forcing dev to prod promotion on interactive work is just operational theater. So, two practical principles follow. ClickOps is fine, but only in the interactive tenant and the dev environments.
[08:32] But in higher environments, only service principal deployments are allowed. And this is what the tenancy model looks like in terms of Databricks primitives, workspaces, Unity Catalog,
[08:48] identity management. The tenancy model gives you boundaries, uh but boundaries aren't a deployment lifecycle. The interactive tenant flow is the easy half. Provision an AD group, grant read
[09:04] access to the shared catalogs, and you're done. The service tenant flow is where the real work lives. Because every service tenant is a deployable unit of software, that unit has to traverse dev, then non-prod, then prod. Uh
[09:19] a service tenant isn't just a service principal and some catalogs. It's a coordinated set of changes across three different planes, governance, schema, and runtime. And they all have to run lands together. So, the fundamental challenge is that those three are deeply related,
[09:37] but most teams deploy them as separate manual steps. A job deploys before its target schema migration has run. A Terraform change adds a schema, but the catalog binding lands in a different pull request a week later. An engineer just runs the migration in
[09:54] prod real quick because the pipeline doesn't handle it cleanly. Every one of those is a partial state outage waiting to happen. The deployment has to treat all three as one thing.
[10:11] So, the first structural choice is the basic one. Where does the code actually live? We made a deliberate choice here. Uh we use a single mono repo for the entire SD2 platform. Every service, all of the infrastructure as code, the shared libraries, the
[10:26] migration files, the bundle definitions, all of it lives in one repository. Um so, let me make those three planes concrete
[10:42] because each one has its own tool, and that's deliberate. Uh the governance plane is Terraform. This is the Unity Catalog scaffolding that defines who's allowed to do what. Catalog and schema management, storage credentials, service principal grants,
[10:57] and workspace bindings, and group permissions. It establishes identities and grants. It does not know about table shape. It does not know about jobs. The schema plane is Alembic. This is the shape of the data expressed
[11:13] as DDL. Table definitions, column changes, primary and foreign key constraints, view definitions. Alembic is a migration tool that normally pairs with SQL Alchemy, but we point it at Databricks SQL. Migration files are version controlled,
[11:28] sequentially applied, and audible for drift. And the runtime plane is Databricks asset bundles. This is the code and configuration that actually executes jobs and workflows, pipelines, SQL warehouses, compute, and notebooks.
[11:44] Three planes, three tools. Uh and they must move together. So because back to the failure mode, a job that runs before its schema exists will fail. A migration that runs before its catalog is provisioned will fail. The dependencies are real, so the
[12:00] deployment has to treat all three as one. So this is the government governance plane in action. Um what you're looking at here is how a catalog gets created in Terraform. Uh the important properties are all on this slide, even if you read no code at
[12:16] all. It's declarative. Um we describe the catalog we want, not the clicks to create it. It's version controlled. It lives in a TF folder of the mono repo. And it goes through pull requests and review like any other change. And it's applied under a service
[12:32] principle scope to the target environment, not by humans clicking around in the UI. This is where the governance plane does its job and nothing more. Terraform provisions the catalogs, the schemas, the external locations, and storage credentials. Then it lays down
[12:47] the grants and the workspace bindings. Who can read, who can write, who can apply tags. What it deliberately does not do is define the shape of any table or to plane deploy any job. It establishes the identities and the permissions and then hands off.
[13:06] The catalog is only the outermost container, right? The Terraform uh once Terraform has provisioned the catalog, it goes on to provision the schemas inside it. And that's what you're looking at here. Uh this is still the governance plane, still Terraform, still running under a service principle scope to the environment we're deploying into.
[13:21] The important thing to understand is that this step uh what this step does and what it deliberately does not do. Terraform establishes the scaffolding, the catalog and the schemas, the containers that the data will eventually live in. It sets up who's allowed to operate inside those containers, but it
[13:36] does not define the shape of any table. And the third piece of the governance plane are grants. This is where the structural boundaries described earlier stop being a diagram and become code. The tenancy model says, "Analysts get
[13:53] this kind of access on this kind of catalog." And this is that rule encoded. When someone asks who has access to what, we don't reverse engineer it out of the platform, we read the file.
[14:10] So now we cross into the second plane, the schema plane. Terraform builds the containers. This is where we define the shape of the data inside it. We do this with Alembic, as I said before. Um So the migration that creates our ETL Here's the migration that creates our ETL logging tables. Each migration has
[14:27] an upgrade that reads SQL templates and executes them through a Databricks adapter and a downgrade that drops those tables. Every migration has an ID and a revises pointer to the one before it. The chain is what makes our schema version controlled, applied in order,
[14:42] and auditable for drift. So the first two planes build the containers and the shape of the data. The third plane is runtime, jobs, pipelines, compute. We define it with Databricks asset bundles. This is a real job definition,
[15:00] the Fusion Core Dimensions Loader, and it's entirely declarative YAML. So the entire runtime footprint of this job, when it runs on what compute, with what arguments, is checked in, reviewed, and deployed like any other code.
[15:15] That's the third and final plane. So we see in three planes, three steps, in this order: uh Terraform for governance, then Alembic for schema, then Databricks asset bundles for runtime. I want to say two things about the sequence.
[15:31] Uh first, the order is not negotiable. Uh a job that deploys before its target schema exists fails the first time it runs. A schema migration that runs before its catalog has been provisioned fails with a permission error. So, rather than ask every engineer to
[15:47] remember that order, we encode it into a pipeline. The dependencies are expressed once in the machinery and not in anyone's head. Second, when I say these deployments are atomic, I do not mean transactional,
[16:02] unfortunately. Uh Terraform does not roll back if Alembic fails. There's no global undo button. What atomic means for us is operational. A deployment either completes fully with a known good state or it fails loudly and visibly.
[16:17] And the next deployment corrects it. We don't roll back. We fix forward and rerun. That sounds like a retreat, but it isn't. It's a stronger than a rollback because rollback hides the failure and item potent fix forward forces you to
[16:33] make the system actually correct. Uh after more than 18 months on this, our failures are still obvious and our time to recovery is still short. So, what orchestrates all this? GitHub Actions.
[16:48] Uh this is a real deploy run, data ingestion into dev, that succeeded in about 16 and 1/2 minutes. You can watch the three planes execute in order right in the pipeline. This is also what makes contributing cheap. Adding a service is just adding a
[17:04] Terraform module, an Alembic migration directory, a bundle, and one workflow entry. From scaffolding to first deployment is a day or two. When we uh we went from three production services to seven in two quarters. And
[17:19] the machinery got easier to use as we grew, not harder, because each new service is just another instance of a pattern that we've already debugged. Now, I want to turn over to my co-presenter Suchit to talk about the rollout of LakeFlow.
[17:45] Good morning, everyone. So, uh, I'm Suchit. I'm the data engineer with the, um, ST2 data engineering team. And, uh, I'm I'm going to talk about, like, how do we get the data into this platform? Now that we have the foundation in place with schemas, with
[18:00] catalogs, and all of those. Now, what we are waiting for is get getting the data into the platform, right? So, uh, how many of you, with a show of hands, tell me, do you remember the keynote from Bilal's keynote where he kind of showed up basically a snapshot of uh, all sorts of
[18:18] tools, which included EMR. And then, basically, it had a glue and many other AWS services just to get the data into the platform, right? So, and it gets pretty complex with, uh, all these tools, many of them doing same thing
[18:33] that it's supposed to do. So, we we had a similar setup within within, as well. What ly standardized on is on an ingestion platform based on a managed ingestion using LakeFlow
[18:50] and a custom ingestion, which is a Python-based ingestion mechanism, basically. So, those are the two mechanisms that we standardized on. And, uh, as Brett was mentioning in the mono repo, ingestion service is one of the
[19:05] service tenants in our tenancy model. So, you'll see a folder within the mono repo, which is a data ingestion service and that gets deployed very similar to all the service tenants that we have. Uh it also goes through the dev life dev
[19:21] life cycle of dev non-prod and prod basically. So, it's one of the service tenants. Now, when we choose an ingestion pattern for assets that we are onboarding or when we are of the applications, uh
[19:36] there are certain considerations that we need to kind of look into. Uh the first one is what kind of source is it bringing the data from? Is it like a seek database? Is it a SAS software like Salesforce,
[19:52] Now, and others? Or is it bringing data from uh files in object stores basically? So, these are some of the uh types of sources. And based on the kind of source, uh types would differ as well. So, like for
[20:08] example, if you're connecting into a SQL server, you would be using uh there are a couple of ways to do this. Like you can use username and password to connect to it. Or you could use uh domain accounts to do an And the why do we need to worry about
[20:26] these considerations is based on these considerations are the ingestion pattern that we use would be different. Like we would use either a managed lake flow injection, it's a username and password based injection. Or if it is NTLM based injection,
[20:42] um there are individual these individual of these patterns have different capabilities. So, we would use that accordingly. And one other con- and also here is what kind of uh does your business require? Do they need
[20:58] a full load every single time or do they need to do incremental loads and things like that? And um also based on your source, whether it's Postgres SQL or SQL Server, there are CDC-based and non-CDC-based kind of ingestion patterns as well.
[21:14] And if you want to store the history once you have ingested, so there are considerations around that as well from the SCD standpoint. Now, let's look at how this ingestion uh pattern works for us. We we already spoke about there are various uh sources where we ingest the
[21:31] data from. And these are the two ingestion patterns that we have built. The one that you're seeing on the top is basically uh repeatable ingestion pattern that we have built using Python uh object-oriented uh framework.
[21:47] Uh and the one in the below is basically the lake flow, which is the managed ingestion. Um and we'll get into the details of that as well. So, based on picking one of these ingestion patterns based on your source, it lands the data into your private catalogs, and
[22:04] only the data platform team has access to this private catalog. So, your business teams and others won't have access to this private catalog. Um and we add a little bit more I mean context to the data like timestamp and things
[22:19] like that, and then publish that data into shared catalogs. That's the catalog which uh basically serves as the bronze layer for your business uh solu- business data products, basically. And from there, you run your ETL
[22:36] um using jobs or workflows and all of those, and then it gets converted into data products, which is your silver and gold layer, basically. And that's where all the business users operate, basically. And I see Kelly smiling.
[22:53] She's one of our product owners here. She uses the data. She's one of the consumers of the data here. So, uh let's look at um the object-oriented Python-based ingestion pattern that we were talking about, which is the custom ingestion
[23:09] pattern. Let's take a quick deep dive on that one. Um source coordinator is the mechanism which we use to connect to our source system. So, this is where you would tell what the connection strings are and what
[23:25] are the credentials to connect into the source systems. And you would also define the objects that you want to bring from source systems into uh Databricks, basically. So, tables um and the keys associated with the tables and so on and so forth.
[23:42] And the last one that you see is basically the logic for the coordination. Now, um let's move on to the extractor. This is the actual logic to ingest the data from source to Delta tables for our
[23:59] Python-based ingestion model, basically. Um now, what I mean many of you might be thinking why why this particular Python-based pattern, right? So, this is a reusable ingestion pattern. So, instead of building individual
[24:15] implementations for each ingestion of your business applications, this is a reusable pattern which you can apply across Let's say similar source, DynamoDB, putting the data into S3 for a business, and
[24:30] eventually uh loading it from that S3 into Databricks. So, in that scenario, that might be a scenario which other businesses might have as well. So, this is a reusable pattern that work across your uh businesses, and you're not repeating the code in this scenario.
[24:48] Uh lastly, this adapter is how the Python-based model acts to your source as logic. In this case, AWS S3 connector is what you're seeing. Uh adapter is
[25:04] what you're seeing here. Um Let's uh one thing which I wanted to mention on the adapter piece is this is how the domain language that we, the data engineering team, speak internally.
[25:20] Adapter, coordinator, and the extractor for the Python-based ingestion, right? So, let's move on to So, those are things for the capabilities where you need more uh granular control and things like that. Now, let's move on to the managed
[25:37] ingestion piece, right? So, we use uh Lake Flow Connect for managed ingestion. And for if folks if you are um not familiar with Lake Flow Connect or haven't used one, Lake Flow Connect is a managed offering from Databricks for
[25:54] ingestion from over 100 plus data sources. They have multiple connectors built um for Lake Flow Connect. All the way from Databricks systems to uh SaaS softwares and things like that. And the structure of a Lake Flow Connect
[26:11] is basically a gateway and a pipeline. So, just to kind of uh split that further, what gateway does is it connects into your source system, it gets the data from the source system, brings it to a volume,
[26:26] which is a staging area within Databricks. So, depending upon source that you're connecting, it would use CDC and things like that. Would connect to that, bring it to the volume, and that's your staging area. And then pipeline is the second portion of the gateway. What
[26:41] it does is it takes the data from the um from the the staging area volumes and pushes it into your Delta table. So, that's how your data gets into your Delta table. So, now um And I wanted to mention this. Lake Flow
[26:59] in the background uses Spark declarative pipelines to actually get that data. What what it does for the ingestion is it's like it basically brings millions of records in seconds into the Delta table basically.
[27:18] So, um what do we use on the runtime plane uh that Brett was referring to for delivering Lake Flow, we use Databricks asset bundles. And what you're seeing here is an implementation of gateway and the pipeline for one of the actual uh
[27:35] similar the same example business that we're talking about the fusion uh what we have built for ingesting data from this is I believe is an example of a PostgreSQL SQL implementation basically.
[27:52] So, let's see what does this pattern or framework enable us basically. So, it helps you onboard your businesses faster. And in future, you can also use Genie agents and things like that to onboard your businesses to
[28:07] create these um catalog schemas as part of the onboarding process. So, this consider this framework as a foundation to get you to that particular maturity level in terms of automation using agents basically. Um secondly,
[28:24] it gives you an ontology of services for with business aware context to it. So, like uh you saw in uh when Brett was sharing the mono repo, you had different services for different businesses and things like that. So, that gives you an
[28:39] ontology of how business services are structured. So, that way um if you want to just go back to the repo and understand which business is using what service and things like that. it It gives you that specific context. Now, the third one is my favorite. Uh
[28:55] it expedites the data ingestion uh into the platforms. From months, we went to days in terms of ingesting data into the platform. And what's cool about this particular pattern is
[29:11] it will enable you to go to minutes as well with agents enabling using this framework, basically. Now, let me clarify that. I mean the implementation of ingestion is what will go from uh day months to minutes to
[29:29] uh months to minutes. It's not the ingestion itself. That depends on your data, volume of the data, and things like that, right? So, um the other thing which I think you guys might have heard during the keynote as well is auto healing of your operations, the
[29:45] Genie Zero Ops that you heard, right? This pattern enables you to kind of fix your jobs and pipelines on the go, and Genie Ops can kind of connect to this to enable you with that auto healing. So, it it it's basically
[30:01] in terms of uh enabling feature for Genie Ops. Lastly, I mean, end-to-end governance and control across your businesses, across your data, and all of those is something that this pattern um brings in. Let me Let me bring it all together,
[30:16] right? I mean, what what does this framework do for you? It gives you predictably predictability, repeatability, and reusability across your code base that enables the agentic vision and agentic future that you guys are looking to build for for
[30:33] your businesses, basically. So, with that, I think that brings me to the end of the talk here. So, I would like to invite Brett uh back over for any questions that you guys might have.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.