Predictable Migrations: Scaling Lakebase Adoption With Provider-Agnostic Readiness Tools

Summary

The Lakebase Advisor is a Databricks application that performs automated pre-migration assessments for Postgres workloads, analyzing schemas, extensions, configuration limits, and version disparities to generate a compatibility roadmap before migration begins.
AI agents require millisecond-speed, instantly provisionable database infrastructure, and Lakebase addresses this through compute-storage separation, branching, autoscaling, and scale-to-zero capabilities unavailable in traditional managed Postgres services.
The readiness framework classifies migration findings into blockers, warnings, and compatible-as-is categories, allowing architects to estimate refactoring effort precisely and build a high-integrity execution plan rather than discovering issues post-migration.

Predictable Migrations: Scaling Lakebase Adoption With Provider-Agnostic Readiness Tools

Successful migration from managed OLTP services to Lakebase depends on one factor: identifying architectural nuances before the transition begins. Instead of a “move-and-fix” approach, teams need a rigorous, automated way to audit environments and quantify technical debt. this video introduces a unified readiness assessment framework designed to evaluate any managed OLTP workload for Lakebase compatibility. We will explore how an automated engine interrogates database metadata to audit schemas, extensions, and configuration limits. By analyzing version disparities and provider-specific constraints, the tool generates a high-fidelity compatibility roadmap. This approach allows architects to identify blockers early, estimate refactoring with precision and build a high-integrity execution plan. Attendees will learn to standardize pre-migration checks, transforming a high-stakes transition into a predictable, well-orchestrated architectural upgrade.
Talk By: Fatine Boujnouni, Senior Specialist Solution Engineer DWH, Databricks ; Savannah Longoria, Senior Developer Advocate, Databricks ;

Chapters

FAQs

What is the Lakebase Advisor?

The Lakebase Advisor is a Databricks application built by Databricks solution engineers that automates pre-migration readiness assessments for Postgres OLTP workloads migrating to Lakebase. It interrogates database metadata to audit schemas, extensions, and configurations, classifying findings as blockers, warnings, or compatible-as-is to produce a high-fidelity migration roadmap.

What makes Lakebase suitable for AI agent workloads?

Lakebase provides compute-storage separation, branching, autoscaling, and scale-to-zero capabilities that give AI agents instantly provisionable, isolated database environments. Traditional managed Postgres services can take days to provision and hours to clone, which is incompatible with the millisecond speed at which AI agents need to reason and act independently.

What are the common blockers when migrating from managed Postgres to Lakebase?

Common migration blockers include incompatible extensions, breaking changes across Postgres version boundaries, partial and expression indexes that behave differently, and pgBouncer connection pooling requirements. The Lakebase Advisor surfaces these as classified findings before migration begins, allowing architects to estimate refactoring effort and avoid post-migration surprises.

Why do Postgres migrations often fail without a prior readiness assessment?

Postgres migrations from managed services frequently fail because teams underestimate hidden costs from undetected configuration changes, extension incompatibilities, and schema gotchas that only surface in production. A 'move-and-fix' approach is risky for OLTP databases backing business-critical applications; the Lakebase Advisor transforms a high-stakes transition into a predictable, well-orchestrated upgrade.

Full transcript

[00:09] So, welcome everyone. We're really glad you made it because we admit that the title of our talk is very long. So, we weren't sure you will made it to the register button, but here you are. You found the room, you sit down. So, we're really glad you're here.
[00:23] So, I'm Fatin. I'm a senior specialist solution engineer based in Paris, and I'm really glad to be here today with Savanna to talk about OLTP migrations. I will let Savanna present herself.
[00:36] >> Hi, I'm Savanna. I was previously a field engineer, which kind of inspired this application that Fatin and I worked on. Now, I'm a senior developer advocate here at Databricks.
[00:50] >> So, we just wanted to make it clear. Today, we're going to talk about OLTP migration and more specifically Postgres migration. So, those are the databases that are behind
[01:02] your most business critical information in the company. So, those are the ones that are behind your web application, your mobile application, and your operational
[01:16] workload. So, moving them from a system to another isn't just moving tables, it's also moving business processes that everybody in the company care about. And that's why we're going to be talking
[01:29] about this today. So, to set the context, so as we all know, AI workloads today are creating new expectation for the infrastructure.
[01:44] Because the product teams want to move faster with the those new AI workloads, the developers want environments on demand for their AI agents, and the business teams want AI-powered workflows
[01:56] without waiting ages for the infrastructure. But the OLTP database underneath those workflows is still working in the traditional way. So, if you think about it, like today
[02:08] your application can scale in seconds and your AI agents can spin up in seconds, but the database underneath still have like still take days to provision and hours to clone and like
[02:21] this is exactly what we try to resolve with LakeBase. Alitsavena presents what LakeBase is. But before >>
[02:34] about the infrastructure around Postgres and why it's not following up like the agent speed and scale. So, today the agents move at millisecond speed, which
[02:46] is different than the infrastructure underneath. Here on on the left, we stated what the AI agents Iran needs today. Our agents need their own environments, their own isolated environments because
[02:59] they need to reason, they need to act, they need to evolve without touching the context of another agents. And their experiments they last for days, not quarters, so they need an infrastructure
[03:13] that is that can spin up quickly and that we can also destroy quickly. And the infra life cycle of the those AI agents need to be aligned with the
[03:25] workload life cycle. We don't want like to provision instances and and don't destroy them and leave them there. And the architecture today doesn't
[03:38] align with the AI agents Iran needs because today when we provision a database for your AI agent, that means cloning the database and copying terabytes of data.
[03:50] Also, as we multi- multiply the AI agent experiments, the instances um accumulate, and nobody is here to fully decommission them, which means the
[04:02] complexity and the cost will grow in the background for your company. And also, we notice that the compute uh runs even where your AI agents doesn't run, which means again, more cost that
[04:14] grows in the background. And at the end, like, in order to provision a a database instance today, you need hours to provision, you need a change request, etc.
[04:26] And yeah, we want to um like, help um move um and evolve this infrastructure with the Lakehouse base um with the Lakehouse solution.
[04:41] >> Now, we're going to talk about Lakehouse. Um so, Lakehouse is fully managed Postgres. It brings serverless Postgres engine into Databricks Lakehouse, making OLTP a first-class citizen along Delta Lake,
[04:54] MLflow, Databricks apps. And I won't give you the entire sales pitch today because I know you've probably heard it multiple times this week um and are very tired. But if you walk away with anything, I want you to walk away um
[05:08] with this. Lakehouse base is Postgres at its core. It's not a proprietary fork, Postgres-like, or Postgres-compatible. It's good old Postgres. But there are some twists um clever twists with
[05:21] Lakehouse base such as the architecture, which disaggregates uh separation of compute and storage. Yeah. So, what I mean by this um is that customer data set
[05:33] doesn't reside on the host that makes Postgres run. Um we have a clear separation between the host that runs Postgres and the software where the data set resides. So, compute and storage are
[05:45] decoupled for cost control, and this separation is what enables features that make it perfect for agent workloads, such as copy-on-write branching, auto scaling, and scale to zero.
[06:00] And the goal of the talk today isn't to convince you to move over to Lake base. Um the goal of this talk is to share our learnings with you about the gotchas that most providers don't like to talk about. Um because often AEs are too busy
[06:12] worrying about a contract rather than what's best for the customer. So, at this end of this talk, we'll show you something called the Lake base advisor. Um and this is something we're building for our field engineering team. We'll also quickly show you a script
[06:25] that we built that you can run yourself if you want to see a holistic view of your database in case you're you yourself are preparing for a Postgres migration.
[06:38] Yeah, so let's talk about why database migrations tend to fail. Um they don't tend to fail at the moment of cutover. They fail in production. What I mean by this is the pattern we see over and over is called move and
[06:50] fix. Teams migrate first, then discover what's broken. And the cost is almost never the migration itself. Um it's the surprises that follow. So, let's say a query that ran 2 milliseconds in the old planner now does
[07:03] a full scan on a table or an extension you forgot um you depended on isn't installed on the new provider. Um these five boxes are um what we're going to talk about today. Um they're
[07:16] different dimensions um such as invisible provider lock-in. So, think features, defaults, behaviors you didn't realize were specific to your old host. Um we have schema assumptions. Uh so, we
[07:29] can think collation, sort order, data type quirks. Constraints that behaved um one way and now are going to behave another way on your new uh destination database.
[07:41] Uh we have configuration drift. Um, so auto vacuum, work mem, um, isolation levels and connection limits, all tuned for the old environment and are wrong for the new one.
[07:53] And, um, Fatima will also talk about, uh, version gaps, um, such as like major Postgres version, uh, stuff that we see come up. But, the point of this is that the migration was never real real risk. It's
[08:06] the unknown, um, it exposed that are. So. >> So, now we'll move to the readiness, uh, talk, which is like the, the goal of
[08:18] our, uh, presentation today. So, uh, readiness is not a simple yes or no question. So, it's more much more nuanced than that, and it's a spectrum, um, around several dimensions.
[08:32] Um, and answering the readiness question would help answer, uh, different question for different stakeholders. So, the business stakeholder readiness help answering a practical question. So, um,
[08:45] is this migration would be, uh, light lift, or does it require a moderate transformation? So, this help him understand what he's signing up for in terms of cost, time, and disruption.
[08:59] And for a business stakeholders or a DBA, the the question is different. The question is around where the risk is coming from. Is it from a schema change? Is it from a version gap? Or is it from
[09:11] an extension that doesn't exist on the target system? So, this is why we're talking about, um, how to assess this readiness today. So, we came up here with a framework around five dimensions to help you
[09:25] assess and also to share with you how we do it on the field today at Databricks. Uh, the first one is the schema compatibility. The schema compatib- compatibility help us answer the question of can we transfer the data
[09:39] model without losing the semantics. The feature parity is more about the extensions. Like do the extensions and plug-in that we have today on our source system has equivalent on the target.
[09:52] The third dimension we're assessing is the behavioral equivalence and this like we see it especially when we start running queries. We need to know if they
[10:05] are producing the same results under different configurations. The fifth dimension is the operational fit fits. So we need to know if the target
[10:17] supports the connection patterns that we have, the throughputs, the high availability needs. And the last one is the version alignments. Because if we're crossing major version
[10:29] boundaries, we may have some breaking changes. And the thing is not all the issues are equal. So a schema change can take few hours to fix for example. But a missing extension can take days to
[10:42] months if we don't have the replacement on the targets. So we need to um to detect that a bit earlier. And also yeah, so if you know which dimensions
[10:55] are blocking you before starting the migration, this would help you make the whole process more smoother. Savannah will start explaining the first one. We'll go into visit them one by one
[11:09] and then show you how we use that today. >> Yeah, so today our first dimension is schema compatibility. So schema compatibility is where majority of
[11:21] migration projects hit their first wall. Not because Postgres is Postgres, but because most providers tend to have dialect drift. So the sequel compiles fine, it's just it just doesn't behave
[11:34] the same. Um to walk through the places we see this most often, uh data types, so serial and money are considered legacy. Uh so, modern Postgres, including like
[11:46] base, um what identity and explicit numeric uh and copy them straight acr- if you copy them straight across, you'll get a broken sequence, um and ownership and money values that change meaning um
[12:00] within this with the server's locale. Uh it's important to watch JSON and JSON blobs as well. They look identical in DDL, but indexes index and uh indexes compare very
[12:12] differently. Um there's also constraints and indexes. Um
[12:26] um on the new system. And this exclusion constraints um usually need B-tree gist or uh which isn't available anywhere. Um there's also stored uh procs and triggers. So, PL and PGSQL is mostly
[12:41] portable across Postgres versions, um but rarely across engines. So, Oracle, SQL Server, MySQL, and Postgres uh fire triggers in alphabetical ordering. So, ordering assumptions and
[12:54] security definer and invoker quietly change that behavior after cutover. >>
[13:06] urgency vary. So, it gets more and more complex the bigger your database is. Um the next dimension is extensions and provider lock-in. So, extensions aren't just uh installed
[13:20] packages, they're often embedded in queries um and application logic. And this is the number one source of migration blockers that we see today in the field. Um some have full support or
[13:32] partial support or no equivalent at all. Um there's provider specific hooks and hidden dependency chains that make extension risk easy to underestimate. This is why extension mapping must
[13:44] happen before planning timelines. Um a good example that I think comes up a lot, especially um at Neon, is Timescale DB has two extensions. Uh they
[13:57] have Apache 2 version and they have a community uh edition. The Apache 2 version um includes core tables, basic features, and is usually fine to host. The community version um
[14:11] requires a TSL license, but this has compression, continuous aggregates, um retention policies, and incremental refresh. So, it's what people usually want. And our uh like AWS
[14:24] RDS, uh Cloud SQL, Azure Postgres, Neon, um Superbase, and Render, etc., etc. There's a lot of Postgres providers. Uh legally can't offer Timescale um good
[14:37] features without a commercial agreement. So, they either ship um uh Apache only version or skip it entirely or have to find workarounds.
[14:51] >> So, now we're at the third dimension. So, the third dimension we need to watch out for is the version gaps. And this one is tricky because it doesn't appear at the migration level. It's in and it doesn't break in a obvious
[15:04] way. And what makes this tricky is that Postgres upgrade doesn't only come up with new features, it also change some of the behavioral um of the existing
[15:16] like the behavioral the behavior of the existing features. And um so, when we um we change the the Postgres version, we may come up
[15:29] with like we may run into deprecated syntax, for example. We may also encounter different implicit casting. We may We might see also some
[15:42] shifted default parameters. So here we took some examples. If you go from Postgres 12 to Postgres 14, you will find out that the recovery
[15:54] configuration has changed. And also we don't use with IODs anymore. So this is something that came up with Postgres 14. Another example is when we move from
[16:06] Postgres 14 to 16, the local provider has changed to ICU. And we have also some behavioral change on how we sort and compare text. So this may come up in production when you run
[16:20] your queries, you may see some some changes in the behavior of your queries. So what you need to keep in mind is that the version gaps are what we call the
[16:33] silent killers. Because if you do not test them deeply, they may come up on production and they show up on production. That's why we tell our customer to audit for correctness, not just connectivity.
[16:51] The fourth dimension we're assessing is the configurations and the behavioral drift. And what we noticed here is that most of the managed Postgres services impose some provider
[17:04] yeah, provider configuration and some provider limits that are tied to the kind of the instance that you're that you're choosing.
[17:16] So that includes things like the connection limits that is usually tied to the type of the instance you're spinning. And also memory settings, timeout, etc.
[17:28] And the thing is those differences in configuration, again, we don't notice them at the migration phase. We notice them when like the query that's used to take millisecond on the first source is
[17:40] now taking uh, some seconds on the second one. So, that's why we tell our customer to audit for behavior and um, to to see the
[17:52] full settings that they have on the full on the PG uh, settings table. And to assess the ones that are provider defaults and the one that's are user sets parameters
[18:05] uh, before moving moving with the migration. >> And the last dimension is a little bit different than the first four. So, the migration is a once-in-a-lifecycle
[18:17] chance to right-size. You're already paying the operational cost of a cutover. Um, it's important you don't waste it by lifting and shifting your old setup onto the new provider and risk running into the same issues that caused you to
[18:30] migrate in the first place. These um, here on the on the slides are the three signals that pull um, from the source that you should pull from the source before you decide to provision onto the target. So, we have connection
[18:43] patterns. Um, so every idle back back end holds memory. High churn means your pooler is wrong or often missing. Uh, connection pressure is one of the three
[18:55] things that quietly drive CPU up alongside missing indexes um, and lagging auto vacuum. Lake base um, does auto scaling so it sizes compute to the actual peak. Um, so
[19:09] if you have a idle back end um, it doesn't stop costing you as well. Um, query throughput, the the shape of QPS tells you whether you need read replicas, um, where caching actually pays off, and
[19:22] which long-running transactions are pinning the Xim Ryzen, um, and starving auto vacuum. This is also where you'll find queries that have been fine for years, um, but they've been running at
[19:34] 3:00 a.m. and nobody actually noticed that these queries are bad queries. So, lake-based lets you replay workload against, uh, a target shape before cutover. You can have staging and
[19:47] production stay structurally identical, and there's no surprises post migration. Uh, the last one is storage growth. So, bloat is essentially dead bytes on disk. Um, over retain point-in-time recovery
[20:01] is a second storage line item when you're paying for it on top of your live data. So, wall vacuum is a leading indicator of what your new provider, um, will charge you for a backup replication or
[20:13] archive. Um, you can reclaim bloat with vacuum full or PG repack for tables, um, and you can use reindex concurrently for indexes. Uh, we recommend, uh,
[20:25] doing that during the migration window if you can, uh, have downtime. PG repack, um, is a little bit better if you, uh, can't have much downtime. Um, so, yeah, so I we recommend doing
[20:39] that before you start paying a new provider uh, to store the dead bytes. And to make this practical, we automated this entire framework in an internal
[20:51] Databricks app for assessing Postgres to lake-based migrations. We start with the metadata collection, so we capture the shape of the environment, the key configuration signals up front. We also
[21:03] serve We then surface the database profile, um, that gives us a structured view of the source system and from there we run the feasibility analysis across these dimensions to identify where the real
[21:15] migration risks are, not just at the schema level, but also in behavior, configuration, and operational assumptions. So, instead of doing this as a manual checklist, we've turned it into a repeatable process for the field. Um,
[21:30] then I'll go to the So, what I'm showing you now is going to be uh, the script that we created, which is, like I mentioned earlier, the source
[21:43] of inspiration of this app. Um, this is something you can run yourself, um, on the Neon uh, database repo. It's called PG prechecks and it's essentially a
[21:55] heavily modified version of uh, PT PG summary, which is part of the Percona toolkit. And this is something uh, you can run against your production database. We wouldn't be running it for you, so you are able to share um, the information
[22:09] that you want if it's any like security uh, concerns there. Um, we do just recommend that if you are running this uh, to run it like maybe not during peak hours just to be safe,
[22:21] but um, it's been tested against databases with like 300 terabytes, so um, it's pretty consistent. Um, then I'll let Fatin show the app.
[22:33] >> Yes, so um, now we we showed you the dimensions that's where assessing, so you can do it also for like if you want to move to another uh, Postgres uh, provider.
[22:45] >>
[22:58] built that is backed by Lakebase. Um, so this one is assess the feasibility of migrating like the post grace databases to data bricks Lake
[23:10] base. And it started by parsing the metadata bundle and the difference dimensions that we told you about from your source database and evaluating it's across like the five dimensions
[23:23] that we we presented. And then we flag blockers, warning, and what is already compatible. So, how it works? We start by So, it's here. Whenever we want to do new
[23:35] assessments, we choose what is the the managed post grace instance uh we want to assess. So, the script underneath the it's an SH script
[23:47] underneath is the same because we're running in like it's the same script that runs everywhere. Here it just in order to give instructions about what are the prerequisite etc. So,
[23:59] we download the scripts, we share it with the customer, and then he run it on his own database. He can audit the information that are extracted and like decide which one he's more comfortable to share with us.
[24:12] Once we have that, we put the bundle here. So, let's create the demo life.
[24:26] So, we upload it and we arrive to this first screen that show us the profile of the database we're assessing. Here for example, we see that the customer doesn't share the PG version
[24:38] with us. And we have other information on the schema, >>
[24:52] later. We have also all the extensions that are used and how they are used because sometimes we can find extensions that are installed but not used. And then we have some information about
[25:05] the configuration of the server. Once we have all of this, we run the feasibility analysis, and we have a quick overview about what are the blockers, what are the warning, and what
[25:17] is compatible as is. So, the blockers are the points that um cannot allow us to uh migrate as is. So, uh usually it's a missing extension that
[25:30] the customer need to think of before moving to to Lake Base. And um yeah, having this view would allow us also to um to create a migration plan together
[25:42] with the customer and to de-risk this migration. Uh and again, like we're assessing that across the different um the different dimensions that we presented earlier.
[25:55] Once we have a clear vision about it, sometimes we have workarounds, and sometimes we do not. And but like having this report that we share with the customer about the feasibility of him moving his um
[26:10] Postgres to uh to Lake Base is like already helping us advancing with the migration plan. Uh so, that's was it.
[26:22] I guess the last The last thing we want you to ask is to um to join the Lake Base community in order
[26:34] to participate in future paid survey that our product team is suggesting. And that's was it. Thank Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.