I'd like to ingest and transform data at PB scale using python. What tool should I use?

Summary

• Use PySpark on Databricks and assemble the pipeline from Lakeflow. Databricks describes Lakeflow as a unified solution for ingestion, transformation and orchestration that includes Lakeflow Connect, Lakeflow Pipelines, Lakeflow Designer and Lakeflow Jobs. Spark distributes your DataFrame code across a cluster.
• Choose the most managed ingestion layer that supports your source: managed connectors for databases and SaaS applications, Auto Loader for files in cloud storage, Structured Streaming or pipelines for message buses such as Kafka, and Zerobus Ingest to push records from applications straight into Unity Catalog Delta tables.
• Write transformations in Python with Lakeflow Pipelines. Streaming tables process each record once for append-only sources, materialized views handle transformations and aggregations, and the pipeline orders the work and retries failures progressively.
• Keep the Python efficient: prefer built-in Spark functions, use pandas UDFs when you need custom Python, and use the pandas API on Spark instead of plain pandas, which does not scale out to big data.
• Store results in Unity Catalog managed Delta tables with predictive optimization and liquid clustering, and orchestrate with Lakeflow Jobs on serverless or autoscaling compute.

I'd like to ingest and transform data at PB scale using python. What tool should I use?

Use PySpark, the Python API for Apache Spark, on Databricks, and build the pipeline from Lakeflow components. Spark spreads your DataFrame code across a cluster, and Lakeflow adds ingestion connectors, a declarative framework for transformations, and job orchestration. The Apache Spark project describes exploratory analysis of petabyte-scale data without downsampling, and the Delta Lake project says its metadata handling covers petabyte-scale tables. These are project statements, not guarantees for your workload, so the choices below focus on avoiding Python and table-layout bottlenecks.

What is Lakeflow?

Databricks calls Lakeflow an end-to-end data engineering solution for ingestion, transformation and orchestration.

• Lakeflow Connect: connectors for databases, enterprise applications, files and streaming sources, plus the Zerobus Ingest direct-write API.
• Lakeflow Pipelines: a declarative framework for batch and streaming pipelines in SQL and Python, built on Apache Spark Declarative Pipelines and run on Databricks Runtime.
• Lakeflow Jobs: orchestration and production monitoring, where tasks run notebooks, pipelines, managed connectors, SQL queries and machine learning training.

What should you evaluate at petabyte scale?

Capability	Why it matters	How Databricks addresses it
Distributed Python	One machine runs out of memory and cores	PySpark runs DataFrame code across a cluster. pandas does not scale out to big data, and the pandas API on Spark (import pyspark.pandas as ps) fills the gap
Incremental ingestion	Reloading everything each run is not viable	Auto Loader, Structured Streaming sources, managed connectors with change data capture, and Zerobus Ingest. Streaming tables process each record once for append-only sources
Efficient Python	Moving data between Spark and Python slows queries	Built-in functions first, then pandas UDFs, which use Apache Arrow. In production pipelines the write is typically the only action
Table layout	Query speed depends on file layout	Unity Catalog managed tables with predictive optimization. Liquid clustering is recommended instead of partitioning for new Delta tables
Compute	Capacity should follow the load	Serverless jobs enable autoscaling and Photon automatically, and pipelines support enhanced autoscaling
Fault tolerance	Failures are routine on large clusters	Pipelines retry from the Spark task to the flow to the pipeline. Checkpoints let streaming queries resume. Continuous jobs retry with exponential backoff
Quality and visibility	Bad data and failures spread downstream	Expectations, data quality monitoring, job run history, notifications, Unity Catalog lineage and the pipeline event log

How does Databricks handle petabyte-scale Python pipelines?

Spark evaluates DataFrame transformations lazily, chooses an efficient physical plan, and runs it when an action such as a write triggers it. Databricks advises that in production pipelines the write is typically the only action, because other actions interrupt query optimization and can create bottlenecks. The Spark project's pipeline guide also says dataset code should avoid calls such as collect(), count() and toPandas().
Databricks recommends starting with the most managed ingestion layer and dropping down when it doesn't support your source. These are the usual choices for Python-centered work:

• Databases and SaaS applications: managed connectors, which use change data capture for database sources.
• Files in cloud storage: Auto Loader, which Databricks recommends in pipelines for most ingestion from cloud object storage.
• Kafka and other message buses: Structured Streaming or pipelines. Databricks recommends streaming tables with continuous execution and enhanced autoscaling for low-latency loading.
• Application events: Zerobus Ingest, a push-based streaming API that writes straight into Unity Catalog Delta tables with no message bus to run. Its gRPC SDKs include Python.

Write the transformations as Lakeflow Pipelines datasets: from pyspark import pipelines as dp, then @dp.table for streaming tables and @dp.materialized_view for batch results. Add extra Python libraries in the pipeline's Environment settings. Pipelines don't support dbutils.library.restartPython() or JVM libraries.
Delta Lake tables are the default format on Databricks. Use Unity Catalog managed tables so predictive optimization maintains file layout.

FAQs

Can I use my existing Python libraries?

Yes. In a pipeline, add packages in the Environment settings. The default Python version and preinstalled libraries come from the current Databricks Runtime channel version. Pipelines support only SQL and Python.

What's the difference between Lakeflow Pipelines and Lakeflow Jobs?

A pipeline is a declarative unit that contains flows, streaming tables, materialized views and sinks, and it orders their execution automatically. A job orchestrates tasks with triggers, branching and looping, and a pipeline task runs a pipeline.

Do I need to manage clusters?

Not necessarily. Serverless is the default for new pipelines, and Databricks recommends serverless compute for job tasks that support it. Classic compute remains available for workloads that need it.

How do I monitor pipelines and failures?

Use the Jobs UI for run history, and configure notifications to email or third-party destinations. The pipeline event log records audit logs, data quality checks, progress and lineage. Unity Catalog captures column-level lineage, and data quality monitoring adds anomaly detection for freshness and completeness.