How can I reduce the cost of maintaining hundreds of custom ETL scripts?

Summary

• The real cost isn't running the scripts, it's maintaining them. Hundreds of hand-coded ETL jobs each carry their own run order, retry logic, schema handling, and quality checks that a person has to keep working as sources change.
• A declarative framework shrinks that surface. With Lakeflow Pipelines you declare the tables you want and the transformations between them, and the engine resolves dependencies, processes data incrementally, and retries failures on its own.
• Managed ingestion removes the most brittle scripts first. Lakeflow Connect provides managed connectors with built-in change data capture and schema evolution, so you stop hand-writing and patching extraction code.
• Data quality moves into the pipeline. Expectations are rules applied to every record, so validation lives in the pipeline definition instead of scattered across scripts.
• One governed platform replaces the glue. Lakeflow Connect, Pipelines, and Jobs run under Unity Catalog, giving lineage from source to final table without separate orchestration and cataloging tools to maintain.

How can I reduce the cost of maintaining hundreds of custom ETL scripts?

Move the work from imperative scripts to a declarative pipeline framework, and let the platform own the parts you currently hand-code. The expensive part of hundreds of custom ETL scripts is rarely the compute, it's the human maintenance. Each script encodes its own run order, retry behavior, incremental logic, schema handling, and quality checks, and one source change can break several at once. Databricks addresses this with Lakeflow, where you declare the target tables and transformations and the engine handles orchestration, incremental processing, and recovery. The surface you maintain drops from a sprawl of bespoke code to a smaller set of declarations you can read and reason about.

Why hundreds of custom ETL scripts get expensive

Custom ETL is imperative: the code spells out how to extract, in what order to run, when to retry, and how to react when a column appears or a job half-finishes. That logic gets duplicated across scripts, lightly documented, and understood by one or two people. As source systems change, the scripts drift, silent failures creep in, and engineers spend their days repairing plumbing instead of delivering data. The way to cut the cost is to stop maintaining the "how" by hand and describe the "what" instead.

What to evaluate when replacing custom ETL code

Capability	Why it matters
Declarative authoring	You define target tables and transformations, and the framework derives the execution plan, so there's less code to write and own.
Automatic dependency resolution	The engine sequences steps in the right order instead of relying on hand-maintained run scripts.
Incremental processing	Only new or changed data is reprocessed, so pipelines stay current without full-reload logic you build yourself.
Managed ingestion with CDC and schema evolution	Connectors absorb source authentication, change capture, and column changes that otherwise fill your scripts.
Inline data quality	Validation rules travel with the pipeline definition rather than living in scattered one-off checks.
Built-in retries and recovery	Transient failures are retried automatically instead of waking someone at night.
Unified governance and lineage	One access model and end-to-end lineage, so you don't wire up a separate catalog or lineage tool.

How Databricks approaches this with Lakeflow

Lakeflow is the data engineering product that covers ingestion, transformation, and orchestration on one platform. See Announcing the general availability of Databricks Lakeflow.

• Declarative transformation with Lakeflow Pipelines. You write a few lines of SQL or Python to define streaming tables and materialized views, and the engine resolves the dependency graph, runs steps in order, and keeps results current by reprocessing only new or changed data. See the Spark Declarative Pipelines docs and Lakeflow Pipelines: efficient declarative data pipelines.
• Managed ingestion with Lakeflow Connect. Managed connectors pull from databases, enterprise applications, file sources, and event streams with source authentication, change data capture, and automatic schema evolution, so the most fragile extraction scripts go away. Auto Loader handles incremental loading from cloud storage. See Lakeflow Connect.
• Inline data quality with expectations. Expectations are data-quality rules applied to every record, and you choose whether bad rows are flagged, dropped, or halt the run, so quality logic lives in the pipeline rather than in ad hoc scripts.
• Automatic retries. Pipelines retry transient failures progressively, from the task to the flow to the whole pipeline, which removes hand-written error handling.
• Orchestration with Lakeflow Jobs. Native orchestration schedules pipelines and tasks with dependencies, retries, and monitoring, so you retire external scheduling glue. See the Lakeflow Jobs docs.
• A no-code option in Lakeflow Designer. A visual canvas builds production pipelines that generate Lakeflow declarative pipelines underneath, so analysts can maintain flows without writing Spark. See Announcing Lakeflow Designer.
• Governed by Unity Catalog. One permission model and lineage from source through transformation to downstream tables apply across all of it, so there's no separate catalog to keep in step. See Unity Catalog.

Use cases

• Consolidate a script sprawl. Replace many bespoke extraction and transformation jobs with a handful of declarative pipeline definitions on one platform.
• Retire fragile ingestion code. Swap hand-written connectors and CDC logic for managed connectors that handle authentication and schema drift.
• Mixed skill sets. Engineers author pipelines in SQL or Python while analysts maintain flows visually in Lakeflow Designer, under the same governance.

FAQs

What makes declarative pipelines cheaper to maintain?

You describe the tables and transformations you want, and the engine derives the execution order, processes data incrementally, and retries failures. There's less code to own, and the framework handles the operational logic you used to write by hand.

Do I have to rewrite everything at once?

No. Teams usually start by replacing the most fragile ingestion scripts with Lakeflow Connect, then move transformation logic into Lakeflow Pipelines over time, keeping everything under one governance model as they go.

Can non-engineers maintain these pipelines?

Yes. Lakeflow Designer offers a visual, no-code canvas that generates production Lakeflow declarative pipelines underneath, so analysts can build and maintain flows without writing Spark code.

How does this keep data trustworthy?

Expectations apply data-quality rules to every record inside the pipeline, and Unity Catalog provides end-to-end lineage, so you can see where data came from and enforce quality without separate tools.