First-Class CDC Support in Spark 4.2

Summary

Apache Spark 4.2 introduces a unified CHANGES API that handles CDC across SQL, DataFrames, and streaming for any Data Source V2 connector, replacing three separate per-format CDC APIs that previously existed for Delta, Iceberg, and Hudi.
Delta Lake switches to read-time CDC by default, computing change data from row IDs and commit versions at read time rather than writing change files during writes, eliminating a 1.2x to 2x write overhead that analysis of 1.5 billion CDC calls showed was often unnecessary.
Data showing that 70% of those 1.5 billion CDC calls had no CDC files read validated the shift to read-time CDC as the correct default, and this video includes a live demo of the new CHANGES SQL syntax for inserts, updates, deletes, and net changes.

First-Class CDC Support in Spark 4.2

Apache Spark 4.2 makes Change Data Capture (CDC) a first-class engine feature. A unified CHANGES API now handles batch and streaming across SQL and DataFrames for any Data Source V2 connector—eliminating per-format dialects. Building on this, Delta Lake delivers read-time CDC by default, requiring zero stored change files. Join us for a deep dive and live demo.
Talk By: Gengliang Wang, Staff Software Engineer, Databricks ; Johan Lasperas, Staff Software Engineer, Databricks ;

Chapters

FAQs

What is the new unified CHANGES API in Apache Spark 4.2?

The CHANGES API is a format-agnostic way to query change data capture in Spark 4.2, returning the same output columns whether used via DataFrame, SQL, or streaming across any Data Source V2 connector. It replaces three separate, incompatible CDC APIs that Delta, Iceberg, and Hudi each maintained independently.

Why is Delta Lake switching to read-time CDC by default in Spark 4.2?

Write-time CDC in Delta required storing change files at write time, adding a 1.2x to 2x overhead on merge and update operations. Data showing that 70% of CDC calls across 1.5 billion invocations had no CDC files read confirmed that most of this write overhead was being paid unnecessarily, making read-time computation the better default.

How does Delta Lake achieve CDC at read time without write overhead?

Delta Lake's read-time CDC works by using row IDs and commit version metadata already present in the transaction log to reconstruct change data on demand at read time. This means any engine can read CDC without needing write-time change files, and there is no longer a requirement for the table owner to explicitly opt in.

What does the live demo in this video show for the new CDC syntax?

The live demo shows the new CHANGES SQL syntax in action, demonstrating insert, update, and delete events returned with labeled rows, and the net changes option that collapses intermediate states to show only the final result per key. The demo covers both batch and streaming use of the CHANGES API on a Delta table.

Full transcript

[00:08] Good morning everyone. Thanks for being here. I'm excited about this one. Here's the whole talk in one line. SPAT now has first class CDC support in SPA 4.2 and Delta will use it to turn on new time
[00:22] CDC on by default. A quick hello. I'm Gang Wang, an APA spark committer and PNC member at data bricks. This is join work with my
[00:35] colleague Yan. Yan cannot be here today but he view much of this with me.
[00:48] Change data capture CDC is just the list of low changes in a table. If you know data change data free is the same idea to keep it simple we'll say CDC for everything today
[01:07] here is the my ship and it's a fun one stop seeing table a table as a static snapshot instead ask a question what has changed between two pointers of time every change law comes back with a label
[01:20] insert delete or update. That's it. Let's look at one example using the new SPA syntax will show today.
[01:34] All the 101 get added two inserts. Then one one goes from place to shipped. You you see the old lows and the new
[01:46] lows and then all the 102 is deleted.
[01:58] And here is why it matters. CDC runs everywhere every day. On the left, open source jobs, copy data, keep history, sync system. On the right, datab bricks
[02:11] products,
[02:23] the blow bearing infa for both OLAP and OLP.
[02:37] Still two problems with CDC today. Proble one every forman builds its own CDC. Delta has a table value function. Iceber has a procedure hoodie and option.
[02:51] The same idea, three APIs, three sets of divorce to learn and spark the engine never has CDC support.
[03:03] So each format has to fill the gaps on its own. Problem two is the cause delta today the the default delta today
[03:18] is lighttime CDC and to be fair it has a real upside loss because the change computed up front
[03:30] but the loyer pays for it about 1.2x 2x on the merge and update
[03:42] and to lo those CDC files every engine has to increment the writer. Der DB for instance cannot write such CDC files for data lake and plus only the table owner can turn
[03:54] it on time freaks all that no H fire any engine can write and it's owner by default the reader does
[04:08] the computing from 1x to 1.x X of the lead time for heavy updates. The next slides will show the data.
[04:24] So let's be honest about the course. This is right moment. If the table contains and delete only the list speed are identical. If the heavy is up the update heavy up
[04:38] to about 1.5x for very big commits more work but here is the thing the majority of the change fees are never read at all
[04:56] let's look at the number we pulled yesterday there are 1.5 billion CDC Louis over the last 60 days in data fabrics for 74% of them
[05:10] the lynch has no CDC files at all which means that the Louis time and right time are identical and for the other 26%
[05:22] touches update where the right time log the CDC files uh up front and also we we observe a patent
[05:35] about um 30 to 40% of the LE is asking a lens which is computer earlier in the same day. So we can actually build case for this
[05:56] and so here is the fix and it's simple 4.2 tool ships first class CDC support one syntax one algorithm one small content contract let's walk through each
[06:16] the new is just one word changes as the changes between two versions or two time steps it works for streaming too and it works for any sources is
[06:29] that's problem one solved normal performat dialects same feature data frame form use do changes API
[06:41] read the stream also give you the um streaming version the output column are identical to S code so it's just a preance between um SQL and data frame um
[06:53] sand engine underneath and and is no data only. You can implement the same contract in iceber hoodie and so on.
[07:10] And this is the whole connector API and it's tiny dot columns details the output schema basically is the the data schema plus
[07:22] three metadata columns and a scan builder list the low change loads from the connector low ID and low version is just named the identity and version columns.
[07:36] And that's the whole job. The hard part pairing deletes with insert duplication computing updates computing net changes stays in spark connector used to
[07:50] implement all of these. Now it's just a single rapper. And to make it even more efficient, there are three yes or no flag. And for
[08:03] each one the connector only decurs what you already did. Carry over duplication compute updates and spar does the less. For
[08:15] example, if the table is with delion vector, it can flag the first one as force and spa won't um postprocessing for the dduplication.
[08:32] And here is a algorithm um sparse for re over the lo stream partitioned by low id and commit version. If a low has delete and insert
[08:44] as a pair in the same commit then it's a real update we will spot will label them as um update pre- image and update post image.
[09:01] And the second relay um for computing net changes we we use a window operator per low all the by the commit version and then we just keep the first and the
[09:14] last event and label them to the net whether it's a single insert a single delete or after is per low and notice that both D are using the
[09:28] existing SPA operators so that you know they they can leverage the existing whole stage cogen adaptive query execution and so on and also it
[09:41] will works on vector engines like fun kami and so on
[09:55] yeah so um reading the fe is just half the The other hub is applying it and spat that too. The syntax create for auto CDC into a target key says which column
[10:08] identifies the low sequence by orders the events. So lay all out of all the records will still be landed correctly.
[10:27] One level up auto CDC is a new SDP feature in Spark 4.2. You declare the data set and the flow sparse handles the dependency, checkpoints, ordering, retries
[10:40] and so this plug into a manage pipeline instead of one of script. If if you want to want a deep dive, you can check the tour auto CDC in Apasa by Jose and Engi.
[11:02] So we have covered the spa engine. Now the data side data is the first connector to implement the new contract and it does it at time instead of right time. So nothing is stored, no extra on.
[11:24] Um this is the headline for data users. CDC becomes available on every data table by default. Nothing to enable, no a table, no table property, no asking for the uh table owner,
[11:38] nothing extra at right time, right? keeps full speed the all item CDC will remain as a op in for special cases
[11:50] there's only one requirement law checking
[12:03] so how does delta p this all for free two columns already keeps for low checking low ID is a stable identity for logical lows updating a low and it will keep the same low ID even though the low is written to
[12:18] a new file it remains the same low commit version is the version where the low last actually changes um for commits like for example like
[12:31] optimize table um the low version is not changed but if there's a real version it will bounce it so Um with the identity and the version we can
[12:44] we can review the whole world change from the log. No H file no new work on the right.
[12:59] Um let's look at one example like what's in the data lake stream and what's um um in the SBA side. So you um for example with deletion on the untouched low are never rewriten so
[13:12] there's no carryovers. So by default spar doesn't do any post-processing and the fe is actually the lost gym. So um spark you know um the connector emits um
[13:29] an insert for any addit or delete for any remove the file. So the for the same string now if the user asks for like updates computing the
[13:43] updates SPA will pair 101 delete and insert into pre pre and post image
[13:58] for the same string. Now ask for net changes. SPA keeps only the end state per low one one next to one insert all the one two is created and then later deleted. So it's gone.
[14:52] Okay. Okay. Oops. I cannot see here in my screen but I will try to make it work. Um
[15:10] >> maybe let me plug and reconnect. >> Mhm. >> I cannot see the O. Okay. and then change
[15:29] >> screen terminal. Okay. Okay. Cool. Yep. So, let's start. We are still using the previous example. We create a table with orders with two columns
[15:43] ID and the status. And as you mentioned the only requirements low checking and we will you know we
[15:57] records order one one and order one one two and then we will update the first order and then remove the second order and now we will demo the new changes
[16:09] syntax. Um we just select um the output from the changes from version one to version three.
[16:21] And you will see that um the you will see the output. We got the first version we insert two records. The second version we
[16:36] uh we update the first order and then drop the second order. And the syntax also supports open-ended um range. So you you want to know what
[16:48] happened after version two, you can just um select you know from all the changes from version two and you will see um see them
[17:00] and as you mentioned SPA supports computer updates in the engine. So if we add this option which compute updates at two you can see that um
[17:15] the the the second version will um the change type becomes update pre- image and update post image which is different from the default
[17:34] and we can you know compute the net change as where as we mentioned um there's you know we will see the end state of like um the two orders order
[17:47] one is like um insert is shifted and then the order two is like um uh insert and deleted so it's gone so you you don't see any uh intermediate changes in
[17:59] the output we can try something different yeah anyway Okay. And as we mentioned, there's no CDC file arrived on the disk. So we should um
[18:14] um let's check check the um table pass and see what's in it. So as you you can see that there's only um data files and data log no CDC file writen everything
[18:29] is on time. Okay. >> Can you do can you do time instead of numbers? >> Uh yes. um times that
[18:48] um say
[19:16] H something's long >> syntax. >> Yeah. Yeah. I think it's I type something long. Yeah. >> Okay. Anyways, support. Mhm. >> Once you figure out the right syntax,
[19:28] you can do it. >> Uh-huh. Uh-huh. Okay. Anyway, >> thank you. Um okay.
[19:45] Yep. Um so two two ideas to take home. Now um SPA has both class CDC support one syntax one algorithm and then one tiny um DSV2 API. And the second is that now delta um will
[20:01] will turn on d time cdc on by d4. You can try on um spa 4 2 delta 4.3 or dbr um 19 public preview.
[20:17] Uh thank you for your time. I would like to take some questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.