Introduction to Lakeflow Spark Declarative Pipelines: Reliable Data Pipelines Made Easy

Summary

Lakeflow Spark Declarative Pipelines (SDP) let engineers declare the data they want to produce while the platform applies proven execution semantics for incrementalization, CDC handling, and backfills automatically.
SDP reduces the complexity of stateful ETL by handling orchestration, checkpointing, deduplication, and incremental processing without requiring teams to hand-write these patterns in every job.
Built-in SCD Type 2 support via apply_changes, data quality expectations, and Genie Code integration for natural-language pipeline authoring make SDP a foundation for agentic data pipeline development.

Introduction to Lakeflow Spark Declarative Pipelines: Reliable Data Pipelines Made Easy

Building data pipelines often means hand-writing complex data engineering patterns like incremental processing, CDC/SCD handling, and backfills in every job. Lakeflow Spark Declarative Pipelines (SDP) are the next evolution in how transformations are written in Spark by default. You declare the data you want to produce, and the platform applies proven execution semantics for you. Built on an open, declarative standard for Spark pipelines, SDP helps teams move faster on hard problems while scaling production best practices consistently. In this video, you’ll see how SDP: Provides a necessary foundation for agentic data pipeline development Reduces the amount of custom logic required for incremental and CDC workloads Makes batch and streaming pipelines easier to build, debug, and evolve over time Standardizes reliability and correctness across pipelines as they scale We’ll walk through concrete examples that show how declarative pipelines simplify otherwise complex data engineering patterns.
Talk By: Zoé Durand, Staff Product Manager, Databricks ;

Chapters

FAQs

What are Lakeflow Spark Declarative Pipelines?

Lakeflow Spark Declarative Pipelines (SDP) are Databricks' next-generation pipeline framework where engineers declare the data they want to produce and the platform handles complex execution semantics like incremental processing, change data capture, and backfills automatically. SDP replaces manually hand-written stateful ETL patterns with a governed, declarative standard for Spark pipelines.

How does SDP handle SCD Type 2 and change data capture?

SDP includes an apply_changes annotation that implements SCD Type 2 history tracking in a single declaration, eliminating the need to write custom merge logic for CDC workloads. This significantly reduces the custom code required to handle slowly changing dimensions and ensures correctness across production pipelines.

Can I test Lakeflow Spark Declarative Pipelines without affecting production data?

Yes — SDP includes a testing framework that lets engineers validate pipeline logic without touching production data, reducing the risk of introducing bugs during development. An event log also provides real-time operational monitoring for running pipelines, making it easier to diagnose issues and track pipeline health.

How does Genie Code work with Spark Declarative Pipelines?

Genie Code can author SDP pipelines from natural language descriptions, generating streaming tables, materialized views, and flows from a plain-English specification. This makes declarative pipeline development accessible to engineers less familiar with SDP syntax and provides a foundation for agentic data pipeline workflows on the Databricks Data and AI platform.

Full transcript

[00:07] Hi everyone. My name is Zoe. I'm a product manager at Databricks. Thank you for being here today. I know it's been a long conference, so thank you for attending. Um I am really happy to be here to deliver
[00:19] this talk on an introduction to Lake Flow Spark declarative pipelines or SDP for short. Um try to keep this engaging as I'm sure you've had a long four days.
[00:38] Everything starts with good data. I'm sure we have a large audience of data engineers and you all know this. All your downstream applications, whether it's generative AI applications your data science teams
[00:51] BI teams, dashboard teams they need good, clean, fresh data. That's where everything starts. But, having fresh, clean data
[01:04] is quite hard. So, let's go through the journey of what that could look like. Um you as a data engineer start out by writing some business logic where you ingest data into some table
[01:17] called a raw data and you read from that, transform it a little bit, and write to a second table. Clean data. Um you play around with the logic and then you're happy. Boom. All the downstream applications
[01:30] can read it. Or rather, it's a little more complicated than that. Um after you play with that transformation logic well you have to start by orchestrating this.
[01:42] You have a bunch of dependencies. Data has to flow from upstream to downstream in a certain order. So, the first thing you need to build is some kind of orchestration or dependency management system. And you then have to
[01:54] maintain that. Next thing that you might have to do is, well, tearing down your table and recreating it every day might not be the most efficient way to do things. It might be
[02:06] costing you a lot of money if your data is very large. So, you think, "Okay, I'm going to start to implement some kind of uh partition system where I'm going to create daily partitions, hourly partitions, and every
[02:19] day I'm going to compute only today's data and maybe also yesterday's cuz maybe some came in a little bit late and now I'm going to have to thread that logic through my orchestration system and kind of embed that in. So, that
[02:31] complicates things a little bit, but so far, so good. Um the next thing is, "Oh, what if a job fails? Like, it failed halfway through. It was kind of processing some data and
[02:43] it hadn't finished processing the whole thing. Well, how do I handle that?" Um now I have to build some kind of checkpointing mechanism to go back and retry and make sure there's no duplicates, that we're not processing
[02:55] data twice, and uh we have to build that logic in. So, things start to get even a little more complicated. Next thing, like, is, "Oh, I'm receiving some bad data from upstream. I'm getting a bunch
[03:08] of null values in the prices and my downstream users don't like that at all. So, now I have to build these checks and every time I receive data, I have to run them and make sure that the bad data
[03:20] doesn't corrupt the the good data downstream and I have to figure out what to do with the bad data. So, I build that into my system as well.
[03:32] Oh, and not everyone can have access to the data that I've produced, even the intermediary tables, so I I better implement governance as well and make sure that um I grant access to only the right people and have policies around that and mask some PII data, so
[03:47] I have to do that as well. My team's complained that the data is not discoverable, so it'd be nice if that were part of the system, too. Oh, and now my downstream team asked me for an extra column
[03:59] in my table, uh because they want to compute a new metric. And they don't want the new metric just starting today. They want the new metric backfilled for the last 3 months. So now I have to implement some kind of backfilling mechanism as well.
[04:13] Uh so I have to build that, and this happens every now and then, so this is kind of an operation I have to handle as well. And every time I push a change, well, I have to make sure I have a version control system, and when I deploy those
[04:25] changes, I have to run some tests first, and it goes through some kind of development, then staging, then production environment. So now we've built a very, very complicated system uh when
[04:38] in the beginning all I was doing was writing some business logic. And what we've noticed is that this operational complexity, this tooling uh everything on the outside of the
[04:50] business logic is dominating in your day-to-day operations. And this is what the Spark Declarative Pipelines SDP product team set out to solve. Is to help you
[05:02] focus your time where you we think you should be spending your time, which is getting value from data. And less on all this operational infrastructure tooling complexity. So what is Lake Flow SDP?
[05:15] Um at a high level, SDP is an ETL framework which uses a simple declarative approach to help you get value from data and build reliable data pipelines.
[05:27] And we have kind of four pillars of a value prop. Um the first one is incrementalization. Uh process only data that is new, and leave everything else untouched. So
[05:39] incrementality is built into the framework. Uh we have unified APIs between batch and streaming. So no matter your use case, whether it's batch or streaming, we process the data in an incremental
[05:51] fashion with SDP. The second one is accelerated speed of development. With SDP, uh whether it's the declarative rather than imperative programming model, whether it's um
[06:04] agents that can help you build pipelines, we'll talk about this a little bit later. Uh we have clearly built into our system an acceleration, and we see that with our customers. They accelerate their time to market for building data pipelines.
[06:18] And the last two pillars are around processing data efficiently, at scale, and automatically managing infrastructure. This is around things like automatic checkpointing, automatic retries, um serverless infrastructure, automatic
[06:32] orchestration, things that you don't have to think about and the framework manages for you. We have thousands of customers using SDP today that uh have proven uh that
[06:44] they're saving time, operations, and cost on building their data pipelines. I invite you to look um we have published a lot of user stories across different verticals, whatever may be most relevant to you,
[06:57] um to check out what they're doing with SDP. And it's not just our customers seeing it. Uh the market has also recognized us uh as a leader uh of leaders in data lakehouses.
[07:09] We are leading the pack in cloud data pipelines, and we are also um at the forefront in real-time stream processing. This was all the high-level things, but
[07:21] I'm sure you've come here to get a little bit more in the details today. So, this is where the fun starts. We're going to be building an SDP pipeline together. We're going to take it step by step, so you're going to get to see what this looks like uh
[07:34] in real life. Uh before we get started, just two concepts that I need you to learn about uh before we dive in deep in the details. The first concept is really like what is
[07:46] a pipeline? You've been talking about these pipelines, but what is a pipeline? A pipeline at the very highest level is this. Um, it's a DAG, um, a graph with a bunch of nodes that are usually
[07:58] tables, but can also be other things. It can be Kafka topics, sinks. And these tables are linked between them with edges or that signify the dependencies.
[08:10] Uh, and this is automatically inferred uh, by our analysis. So, you just write the code um, that defines this and this graph is automatically generated for you and automatically orchestrated for you in the right order and with
[08:23] parallelization wherever it makes sense. So, this is a pipeline. Concept number two is just a programming model. This is what SDP APIs look like. Today, we have two APIs, SQL and Python
[08:35] API, and whatever is available in one is available in the other. So, you don't need to choose. You can also mix and match. At the very highest level in this programming model, you create tables or sinks, but we'll keep it to tables for
[08:48] this example, and you define a kind of transformation, aggregations, joins, etc. Um, that writes into that table. We are also working on a Scala API.
[09:01] Kind of the rumblings uh, of this. So, soon, hopefully three to six more months from now, there'll be a third box here with a Scala API. But, for now, SQL and Python. All right.
[09:14] Let's get to it. This is going to be our use case for the day. We'll keep it simple. Let's pretend we're at a Databricks conference and we're running the merch store. So, we have orders coming in for different items and our
[09:26] finance team has asked us to compute daily metrics. Things like how many items were sold today or yesterday? How many how much revenue did we make? Things like that. So, we're going to build this pipeline in SDP.
[09:40] All right, this is what our target pipeline is going to look like at the end. We're keeping it simple today. Uh we're going to have a first step where we're just ingesting the raw data, all our events, orders, and the paid events.
[09:52] Then, in the next step in the silver layer, we are going to deduplicate and clean up that data, keep only the latest event per order. And in the gold layer, we're going to do that aggregation, that daily aggregation
[10:04] to provide our response to the finance team. How much revenue did we make today? All right. Let's talk about the bronze layer, so the first step in the pipeline. Uh we have a bunch of connectors uh
[10:17] available uh natively inside SDP. Um we you can really ingest data from any or almost any source, whether that's a message bus, things like Kafka, Kinesis,
[10:29] whether that's data from files and cloud storage, uh or databases, or even uh SaaS applications like Salesforce, Workday, SAP, etc. And if you can't find
[10:41] the connector that you need, we also have a bunch of partners uh partner integrations inside Databricks that you can use to get data inside the lakehouse. Um so, you'd pick from these, but in this example, we're keeping it simple.
[10:53] We're ingesting data from files that are in a volume in cloud storage. Um I'm going to be presenting only SQL examples, but again, know that everything's available in Python. Um so, we're reading from a volume uh
[11:07] called raw orders. Uh very, very simple. We're using this nifty little function that we call read files or autoloader uh internally. And what autoloader does is it keeps track of which files have already been ingested and which ones
[11:20] have yet to be ingested, so you don't repeat that logic, and it keeps that checkpoint automatically stored. It does all of that for you. So, every time you run this, only the new files will be ingested.
[11:33] So, we run this. This is what we get. So, we get a bunch of orders in our raw table. Um the left column is the key. So, we have three order IDs. Um for order ID 101, we have a duplicate
[11:46] event. I don't know if you can see it. There's two paid events. Uh for order 102, uh it was created and then it got canceled. You can see it there, canceled. And order 103 was
[11:58] created and then it got paid. So, that's our raw table. All right, let's clean this up a little bit. So, what we want to do now is only keep the latest event for every order ID.
[12:10] And this will help us get to the next step. And And we also want to deduplicate. For this, we're going to use another nifty function that's available inside SDP that we call auto CDC.
[12:23] Auto CDC is a way is a function that takes as input a stream of updates, deletes, and inserts, or change data capture data, and allows you to process it
[12:36] um and store it either keeping only the latest events uh as denoted by the sequence by Here, we're doing it by timestamp, event timestamp, as you can see. Or you can also keep history, track
[12:49] history, something that's known as SCD type two. And with just one simple change, if you shifted that from SCD type one to SCD type two, auto CDC will handle that change for you. Um it's a
[13:02] very very popular function. Um this is like a key reason why many customers adopt SDP. So, we're going to use it. We're only going to keep the latest, so SCD type one in our case, and this is what we're
[13:14] getting. So, now we get only one row per order ID. Um what the first one is paid, the second one canceled, and the third one paid as well. Um and it handles the deduplication.
[13:28] So, we're happy. Um but finance still doesn't have its answer. Uh how many orders did we get uh per item? This is what we're going to do in gold. Now, we're going to write the third layer of our pipeline, and we're going
[13:41] to use We're going to define our table. We're going to group We're going to select an order date, a product, and then some key metrics like counting the orders, the the items sold, and the revenue. And we're going to use a replace where
[13:54] flow for this. If you look at the second line, uh we are saying, "Okay, just only recompute today's data and yesterday's data, and keep everything else in place because we want to be efficient about this."
[14:06] We're going to do this, run this, and then this is what we get. Um because if you remember, the second order got canceled, so it got removed. This is obviously a very very contrived example with only three rows, but this is what
[14:19] we get, and we're able to say to finance, "Well, we sold $60 of hoodies and $30 of mugs yesterday." All right. This is what our final DAG looks like. Uh this is great. We're happy. Or are
[14:32] we? Suddenly, we find that Well, we have a data quality problem, if you recall. Uh we're getting some null prices. We're not happy with that. Uh so, SDP has a built-in data quality
[14:45] framework that we call expectations. This is what the syntax looks like in SQL. You can define constraints on uh individual rows, and I I guess on columns as well, where you can say,
[14:58] "Well, the price shouldn't be null, or the paid timestamp should always be greater than the order created timestamp." Um so, you can define this, and this will be processing as data comes in, as data is in flight. And it
[15:10] the data the data won't even land in the table um before it hits that expectation. So, that that's what's different and new about expectations compared to other data quality frameworks. Um you can also
[15:23] choose the policy that you apply to each expectation. You can choose between three today, fail the pipeline, drop the row, or alert. So, the row still gets written, but you just send an alert.
[15:36] Uh, we are adding a fourth policy soon. Um, quarantine, which has been a feature long requested by customers where you neither drop the row nor write it.
[15:49] You actually route it to a separate table, and this all happens seamlessly inside the engine. So, this is something we're working on right now.
[16:01] All right. This was kind of a focus on the actual programming model and how you can do data processing in SDP, but let's take a step back and look at what the development experience looks like.
[16:14] The cornerstone experience in SDP is our editor. Uh, we ship this to GA this month. So, um, big milestone for us. Um, we have this wonderful
[16:26] cloud-based IDE, as we like to call it, where everything is available in one screen. Everything is available in one UI. So, I'm going to go, you know, or in ordered uh, from one to five, but on
[16:39] the very left-hand side, you can see all the the all the assets that are in the pipeline in a browser. So, you see all your files. Each file usually defines one transformation. Um, and you can
[16:51] choose to structure that the way you want. Um, in the middle of the screen, number two, you have all the actual files that you're editing um, on the fly or that your agent is editing. Uh, we also support multi-tab
[17:04] experience, which is kind of new. Um, on the right-hand side, you see the graph. You see the DAG. So, this is automatically being generated. You're not defining it. You're not clicking through this. This is being analyzed and
[17:17] generated from your code and you see it as you're editing the pipeline. It's right there. At the bottom of the screen, you have the bottom tab, which shows you things There's different tabs there. It shows you things like data previews. So, each
[17:31] table you can kind of get a preview of what the data looks like there. So, you have this nice feedback loop between code, graph, and data previews. Uh and you also have a performance insights tab directly to see how long
[17:44] each run took, um data quality metrics, things like that. So, we wouldn't be in year 2026 without mentioning our integration with Genie
[17:56] code. Uh this obviously integrates with Genie code just like the rest of the platform. Uh today more than 60% of our pipeline code uh is being authored by Genie, which is a
[18:09] almost scary number, uh but it's what's happening today. So, Genie can create, can run your pipelines, can author them, debug them, and manage them. And of
[18:21] course, um this is outside of what we've announced for Genie zero ops, but just simple Genie code just integrates with our pipeline editor and can do all the things for you. And if you're not happy with using Genie code, we also have
[18:33] skills that we've open-sourced, so check those out. Testing. No development experience is complete without testing. This is a functionality
[18:45] that we've recently added. We've added a native Python unit testing and integration testing framework. Uh we have some native APIs, which with with which you can define a mock data
[18:57] set and run um either individual tables and transformation or the entire pipeline end to end. And the key thing here is that it runs in an isolated Spark session uh in a different schema,
[19:10] so it never touches your production data and you can run these tests. How do you run these tests? Today um oh, this is what it looks like. I'm actually not going to go in to too much detail here, but how do you run these tests?
[19:23] Uh you today you have gutter actions that are available in the editor like this. So, the tests are detected by the editor and you can just run them manually like this. Um we are adding a CLI command in the next
[19:36] quarter uh to be able to automate this via CICD as well, if you would like to. You can also generate tests via Genie. You don't even have to write them
[19:48] yourself. Um there's a handy little generate tests button that shows up uh in your various files. So, yeah, check that out. Uh this has been pretty popular.
[20:03] So, we've talked about the experience in the cloud UI, but we also have a plugin that's available in VS Code if you prefer to develop locally. Um we use this in conjunction with something that you might be familiar
[20:16] with already. It's called declarative automation bundles or DABs. Um it is a YAML-based framework uh which kind of defines a project, defines a pipeline, all the files in the
[20:28] pipeline, the configuration and it allows you to define how the pipeline should be structured, how it should be tested and how it should be deployed. And it allows you to deploy pipelines across environments.
[20:46] Speaking of deployment, um well, that's what I just said. Um but you can use DABs to deploy across environment. I guess this graph is a little bit more telling. Uh you can define different workspaces. That's typically how people do it. You have a
[20:58] dev workspace, a staging workspace and a production workspace. Each of them has isolated catalogs usually, so the data is segregated as well, and you would
[21:10] deploy your pipeline to development, check that it works, run all the tests, and then so on and so forth, deploy to higher environments using dabs.
[21:24] So, once you deploy a pipeline, it's also going to be on you to choose the right data freshness for that pipeline. And SDP supports um triggered or scheduled-based pipelines um and continuous pipelines. And this is
[21:37] a setting that you can choose, and you can toggle at any time on your pipeline, depending on your data freshness needs. So, you could decide today that, you know, daily runs is fine, and then tomorrow move to
[21:50] hourly, and then your business user like, "No, we need close to real-time data. So, let's move to continuous." Um this is something that you can choose. Let's talk a little bit about performance, different performance
[22:03] features of SDP. Um some things I'd like to highlight is the tables that are written by SDP are powered by Delta Lake and more recently Apache Iceberg.
[22:15] Um we have features like liquid clustering, deletion vectors, predictive optimization that are turned on with SDP tables that allows you to get the maximum efficiency for your reads and
[22:28] writes. We just released the ability to expose SDP tables as Iceberg in a zero-copy way, so you can get that of those native Iceberg reads from any Iceberg V3 reader.
[22:41] And our query engine Photon is also the un- underlying underlies all the execution for the data transformations. And finally, we have a proprietary incremental MV refresh engine that we
[22:54] call Enzyme, which allows your materialized views to be refreshed incrementally. This is available on serverless. Um a few capabilities that I wanted to highlight was auto liquid clustering.
[23:06] This is not specific to SDP, but obviously integrates well with it as well. Um, so it's this little cluster by auto um, syntax that you see here on screen. This This is a new capability
[23:19] that we introduced maybe a year ago that allows you to get um, to the maximally efficient data layout. It replaces Z-ordering and also just liquid clustering where you have to
[23:31] choose the clustering keys yourself. If you write cluster by auto on a table, the data layout will be optimized by our predictive optimization engine. And you don't even have to choose the clustering
[23:43] keys. And you get maximal um, write and read performance. Photon, we've talked about. Uh, this is uh, the Databricks Spark engine. Uh, it is turned on automatically inside
[23:55] SDP. And you get uh, increased performance for reads and writes. Uh, merges, updates, and deletes have been proven to be much faster with Photon turned on. And you get things also like efficient joins uh, and aggregations.
[24:13] Last but not least, uh, SDP on serverless has been growing tremendously. Uh, the serverless SDP is available both in non-serverless and serverless versions. Uh, the serverless uh, option in SDP provides you with a
[24:25] fully managed versionless product. You don't have to provide any upgrades. Uh, you don't have to up upgrade the Spark version. Um, that just happens automatically. Provides you with some lower latency, fast pipeline startup
[24:38] times. Uh, we have horizontal and vertical auto scaling turned on. And um, I said this before, but our proprietary engine for incrementally refreshing materialized views is available on
[24:50] serverless. And we've implemented another optimization that we call stream pipelining. Um, and that's available on serverless only. And the best thing is that this keeps getting better.
[25:02] Um, if you use SDP, over the years, we keep making performance improvements, and this is something that just benefits all the existing pipelines, {{}quote} {{}unquote} for free, because this we keep making improvements, and you don't need
[25:15] to change anything. You don't need to change your code. As you get the upgrades, it's a versionless product, everything gets faster.
[25:27] Monitoring and alerting. So, we've talked about developing, deploying, but now you want to monitor and alert your pipelines. The most basic building block for monitoring on SDP is what we call the
[25:39] event log. The event log is just a table uh, that gets populated in real time during every pipeline run, and it contains everything. It contains your data
[25:51] quality metrics, your flow execution times, your analysis times, your throughput, your backlog, all those metrics that you want to be monitoring are in the event log. You can use the event log to, well,
[26:05] directly. Uh, you can export that data to the monitoring tool of your choice via something that we call it hooks, which is just a Python script that runs on every event that's created on the
[26:17] event log. You can also power SQL alerts on the event log. And lastly, you can also just build reporting directly with dashboards on top of the event log. So, we have customers
[26:30] doing all three of these or one of these, depending on what their monitoring preferences are. This is what it looks like in the UI. You can also see event log information in the UI, and if you click on one of these events, you would get a detailed
[26:44] um, a detailed pop-up that would have a lot more information. Something we're bringing to life, uh, this quarter uh in a couple weeks is a event logs
[26:57] system table. So, your event logs are per pipeline. With this new system table, all your event logs flow to the same downstream table. This is a feature that has been
[27:09] worked on and also demanded by customers for a while, so we're we have brought it to life. It's coming to private preview uh at the end of this month. So, what this unlocks is observability at scale. You don't have to do things pipeline per
[27:21] pipeline. Um there's no manual config cuz people were kind of hand coding this before. Uh you don't have to miss worry about missing a pipeline. They all flow to the same event log, and we've improved the
[27:34] query ergonomics a little bit by adding variant support for nested structures. Recovery from a failure. As we talked about it, pipelines can fail.
[27:48] Probably will fail at some point. How do we recover from that? And this can be user errors, like you push bad code, or it can really be not at all your fault, like someone pushed bad data from upstream and everything broke because
[28:00] the schema was mismatched. Um and it's not always something you can plan for. We are very happy to announce a new feature that we call streaming time travel, which allows you to recover from bad
[28:13] data or bad code without a full refresh of your pipelines. What streaming time travel does, it allows you to rewind your pipeline, both the data and the checkpoints, to a
[28:25] previously known good state that is consistent between the data and the checkpoints. This is something that's quite hard to do on your own, uh keeping everything consistent if you have 10 streaming dependencies, making
[28:38] sure that you would have to um iterate through your DAG to find a consistent point between those delta versions and your checkpoint versions, and this is all handled for you inside streaming time travel.
[28:50] This is what it looks like. So, you would There's a a drop-down where you would hit rewind. You would get an option to select certain tables inside your pipeline. Then you would see the rewind points
[29:05] that are available for your pipeline, which is to say what points in time, as denoted by a timestamp, can I go back to? You could select one of them. And then you would get a data preview of
[29:17] what your data looked like for each of those tables back in time. And if that looked good, you would hit next, and this would perform the rewind operation.
[29:32] So, now, okay, we've successfully rewound to a previous good state, and we've recovered from our failure. That's great. Let's talk about governance. Um Unity Catalog, of course, is the main
[29:45] governance um product at Databricks, and it seamlessly integrates with SDP. At the very highest level, tables that are written to by SDP lie in a catalog, and inside a schema
[29:58] inside that catalog there is a schema. Inside those schemas there is There are tables. You can grant various permissions on either the table itself, the schema, the catalog. It's a three-level namespace.
[30:11] We're happy to announce, and this is not an SDP-specific launch, but we're happy to announce the GA of ABAC, attribute-based access control, uh which provides a more flexible way of defining governance at
[30:24] scale. Uh the tenets of ABAC are policies. Policies define rules that use attributes. So, in in this example, um the attribute is a governed tag,
[30:36] like hashtag marketing, which could be a column tag or a table tag. Um policies are attached to a scope. In this case, they're attached to a catalog. Um, and so all the tables in that
[30:49] catalog, in that scope, that have that tag will inherit that policy. And what's great about this is that if you create a new if you or someone else creates a new table inside that catalog or in that schema, it will inherit the
[31:02] policy automatically. So, you don't have to worry about every single time going back and granting access or deny uh to to those tables.
[31:17] Um, ABAC offers two kinds of policies today. There's what we call content protection. This is either uh row-level filtering or column masking. Uh, and so you control what uh the users will see in terms of data. They may not
[31:29] have access to certain rows or certain columns. For example, columns tagged as PII. And we have access policies by which you can just grant certain um, certain permissions or deny certain
[31:43] permissions based on, for example, a role. Those are in beta in private preview currently.
[31:55] Last but not least, OSS. Uh, this has been a criticism of SDP over the years and we are happy and proud to announce that Spark declarative pipelines is open source with Spark 4.
[32:07] Um, most major APIs are covered in our OSS release and we are currently working uh cooking up a release of auto CDC, which is probably the last frontier of open uh of APIs that we need to open
[32:19] source. With that, um, what we said at the beginning, everything starts with good data. Well, I would love for you to leave this room
[32:32] and know that good data starts with SDP. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.