How do you get started with Spark Declarative Pipelines?

Summary

• Spark Declarative Pipelines let you define desired data outcomes while the framework handles dependency resolution, execution order, and incremental processing automatically.
• On the Databricks Platform, LakeFlow extends declarative pipelines with serverless compute, a visual editor, and Unity Catalog governance for unified batch and streaming orchestration.
• Following medallion architecture best practices-streaming tables at bronze, materialized views at gold-and adding data quality expectations early prevents bad data from propagating downstream.

How to get started with Spark declarative pipelines

Building production-ready data pipelines typically requires juggling separate batch and streaming workflows, hand-coding execution order, and writing brittle retry logic. When one step breaks, the entire chain stalls, leaving downstream consumers with stale or inconsistent data. According to Gartner, poor data quality costs organizations an average of $12.9 million per year. Investing in data quality management from the start can dramatically reduce these costs.
Spark Declarative Pipelines offer a different approach. You declare what data should exist, where it comes from, and how it updates, and the framework handles dependency resolution, execution order, and parallel processing. This article covers core concepts, setup steps, and best practices.

What makes Spark declarative pipelines different from imperative Spark jobs?

Traditional Spark jobs are imperative. You code each step, read a source, transform, write to a table, and control the execution sequence yourself. Spark Declarative Pipelines invert this model: you describe the desired outcome, and the framework determines how to get there.
Key advantages include:

• Automatic orchestration: Define which datasets should exist and how they are derived. The framework manages dependencies and incremental processing.
• Simplified authoring: Express pipelines as desired outcomes, reducing boilerplate code so teams focus on business logic.
• Batch and streaming in one framework: Cover use cases from cloud storage ingestion to message bus consumption without maintaining separate codebases.

How LakeFlow extends Spark declarative pipelines

Lakeflow provides unified pipelines (batch + streaming) for orchestration, extending Spark Declarative Pipelines on the Databricks Platform. Every pipeline writes to a single, open foundation where data is fresh, consistent, and ready for analytics.
What Lakeflow adds:

• Serverless compute, a visual pipeline editor, ETL pipeline generation with Genie, and deeper Unity Catalog integration including automatic lineage tracking.
• Unity Catalog provides one catalog for all data, managing Delta Lake, Apache Iceberg™, and Parquet with a single set of permissions, lineage, and business definitions that flow into every tool.

Step-by-step: creating your first pipeline

Prerequisites

You need access to a Databricks workspace with Unity Catalog enabled. You also need permission to create a compute resource and a new schema in a catalog.

Pipeline creation workflow

1. Create a pipeline: In your workspace, click New, then ETL pipeline. This opens the pipeline editor with a default name.
2. Define your logic: Use Python decorators or SQL CREATE OR REFRESH statements to declare datasets. The STREAM keyword indicates streaming read semantics.
3. Add data quality rules: Define expectations as SQL boolean constraints. Specify what happens on failure: warn, drop the record, or fail the update.
4. Run and monitor: Click Run pipeline to update the datasets. The editor shows the DAG, data preview, and execution insights.

Structuring medallion architecture with declarative pipelines

Mapping dataset types to the right layer keeps responsibilities clear and pipelines maintainable. Learn more about this pattern in the deep dive on medallion architecture.

Layer	Recommended dataset type	Purpose
Bronze	Streaming tables	Ingest raw data from cloud storage, message buses, or CDC sources
Silver	Streaming tables or materialized views	Clean, filter, enrich with row-level or aggregate transforms
Gold	Materialized views	Pre-compute metrics and summaries for dashboards and reporting

Temporary views are pipeline-scoped and organize transformation logic without materializing data. Use them to break complex queries into manageable steps.

Best practices for declarative pipeline development

• Start small: Build one bronze-to-gold flow end to end before expanding.
• Use expectations early: Catching data quality issues at ingestion prevents bad data from propagating downstream.
• Separate concerns: Keep ingestion, transformation, and aggregation in distinct dataset declarations.
• Test with dry runs: The spark-pipelines dry-run command catches errors without reading or writing data.
• Version control pipeline code: Store Python or SQL definitions in a repository for auditability and collaboration.

FAQs

What are Spark declarative pipelines and how do they work?

Spark Declarative Pipelines is a framework for building reliable data pipelines where you declare the desired state of data and the framework handles execution. On the Databricks Platform, Lakeflow extends this with serverless compute, Unity Catalog governance, and a visual editor.

What are the prerequisites and setup requirements?

You need a Databricks workspace with Unity Catalog enabled, permission to create compute resources, and a target catalog and schema for output tables.

How do you define tables and views using Python or SQL?

In Python, use decorators from the pyspark.pipelines module, @dp.materialized_view, @dp.temporary_view, or @dp.table for streaming tables. In SQL, use CREATE OR REFRESH MATERIALIZED VIEW or CREATE OR REFRESH STREAMING TABLE statements.

What is the difference between materialized views, streaming tables, and views?

Streaming tables handle ingestion and low-latency transformations where each row is processed once. Materialized views pre-compute complex transformations and refresh incrementally. Views exist only within the pipeline and are not persisted to storage.

How do you configure data quality expectations?

Define an expectation as a SQL boolean constraint on a dataset. Specify the failure action: warn, drop the failing record, or fail the entire update.

How do you schedule and orchestrate a pipeline?

Configure a Lakeflow pipeline as a task inside a workflow job. Set a schedule or trigger, and the platform handles execution ordering and retries.

How do you handle incremental processing and streaming sources?

For materialized views, the engine reprocesses only new or changed source data. For streaming sources, declare a streaming table and the framework manages checkpointing and exactly-once processing.

How do you debug and monitor pipeline runs?

The pipeline editor displays the DAG, data previews, and execution diagnostics. Use spark-pipelines dry-run to validate logic without writing data. For ongoing observability of output tables, consider Lakehouse Monitoring.

Can you use Spark declarative pipelines with Unity Catalog?

Yes. Set a target catalog and schema in your pipeline configuration. Output tables are governed with centralized access control, lineage tracking, and the three-level namespace: catalog.schema.table.

What are best practices for structuring medallion architecture?

Use streaming tables at the bronze layer for raw ingestion, streaming tables or materialized views at silver for cleansing, and materialized views at gold for analytics-ready aggregations. Temporary views help organize intermediate logic without persisting data.

Build your first declarative pipeline

Spark Declarative Pipelines remove the need for hand-coded ETL by letting you declare the desired state of data. On the Databricks Platform, Lakeflow runs these pipelines with serverless compute, Unity Catalog governance, and unified batch and streaming orchestration, delivering fresh, governed data ready for analytics and AI.
To go deeper, explore the Spark Declarative Pipelines documentation or try the Databricks self-paced training for hands-on practice. Ready to build? Get started with Spark Declarative Pipelines today.