How do native connectors handle schema evolution, deletes, retries, and data quality?

Summary

• Schema evolution is handled automatically for most SaaS connectors; new columns are ingested on the next run and deleted columns marked as inactive.
• Deletes are tracked via SCD Type 2 (Slowly Changing Dimension) for database and SaaS connectors, automatically adding versioning columns (__START_AT, __END_AT, __CURRENT).
• Automated retry mechanisms with exponential backoff and checkpointing ensure exactly-once semantics; failed pipelines resume from the last checkpoint.
• Data quality enforcement uses a "table-as-contract" model where Delta table schema is authoritative, with rescue columns to catch non-conforming data.
• Lakeflow Connect integrates with Lakehouse Monitoring to track connector health, data quality metrics, and alert when intervention is needed.

How do native connectors handle schema evolution, deletes, retries, and data quality?
Lakeflow Connect native connectors handle the operational realities of data ingestion: source schemas change, records get deleted, pipelines fail, and data quality varies. The platform automates these challenges so you focus on using clean data.

What is managed data ingestion?

Managed data ingestion means the platform handles schema changes, delete tracking, failure recovery, and data quality validation automatically. You define the source connection and target table; the connector handles the rest.

Key capabilities to evaluate

Capability	Why it matters
Automated schema evolution	Source schemas change constantly; the connector detects and applies changes without breaking pipelines.
Change tracking for deletes	Recording when rows are deleted is essential for accurate analytics; deletes must be tracked separately from inserts.
Fault tolerance and retries	Networks fail and source systems go down; the connector must retry intelligently and resume without data loss.
Data quality validation	Schemas enforce a contract; rejecting non-conforming data prevents downstream problems.
Observability and alerting	You need visibility into connector health and when intervention is required.

How Lakeflow Connect approaches each challenge

Schema Evolution
Lakeflow detects schema changes automatically and applies them based on connector type:
For SaaS connectors, new columns ingest on the next run and deleted columns are marked inactive. No reconfiguration needed.
For database connectors (relational databases like PostgreSQL), Lakeflow supports broader changes: adding columns with or without defaults, altering data types, and dropping columns. The connector uses CDC (Change Data Capture) to detect changes and applies them inline.
Handling Deletes
Deletes are tracked using SCD Type 2 (Slowly Changing Dimension), which maintains full history. Lakeflow adds three versioning columns: __START_AT (row effective date), __END_AT (delete date), and __CURRENT (boolean flag).
The behavior varies:

• SaaS connectors with SCD Type 2: Only delete events captured.
• Database connectors with SCD Type 2: Both insert and delete events captured.
• SCD Type 1 or full-refresh: Deletes not tracked; only current state preserved.

Retries and Failure Recovery
Lakeflow implements automated retry mechanisms with exponential backoff. For SaaS connectors, if an API call fails, Lakeflow retries and resumes on the next run using the same cursor.
For database connectors, the system first restarts the gateway for temporary failures. Persistent failures (backwards-incompatible schema changes) flag for user intervention.
Both maintain exactly-once semantics: if a pipeline fails mid-run, the next execution resumes from the last checkpoint with no loss or duplicates.
Data Quality Validation
Lakeflow uses a "table-as-contract" model: the Delta table schema is the authoritative contract, and every record is validated against it.
When a record doesn't fit the schema, Lakeflow rejects it (not silently drops). You can define a rescue column to catch non-conforming records. Lakehouse Monitoring integration provides observability and alerts when quality issues arise.

Use cases

Enterprise data warehouse: Ingests from hundreds of source systems daily. SCD Type 2 tracking captures all changes. Schema changes apply automatically.
Multi-system integration: Combines inventory, CRM, and e-commerce data. SCD Type 2 tracks attribute changes and deletes as historical facts.

FAQs

Do I need to plan for every possible schema change?

No. Lakeflow detects and applies most automatically. You only intervene for backwards-incompatible changes. Lakeflow flags these clearly.

Can I track when specific rows were deleted?

Yes with SCD Type 2. The __END_AT column records the delete date. For SCD Type 1 or full-refresh, only current state is preserved.

What happens if my pipeline fails mid-run?

The pipeline resumes from the last checkpoint on the next run with no data loss or duplicates, thanks to exactly-once semantics.

How do I know when data quality issues happen?

Lakehouse Monitoring tracks schema violations and quality metrics. Set up alerts, and you're notified immediately when non-conforming records arrive.

Can Lakeflow handle different source systems differently?

Yes. SaaS and database connectors have different capabilities. You can mix ingestion modes: SCD Type 1 for slowly-changing tables, SCD Type 2 for audit-trail tables.