How to do feature engineering at scale for classical ML models without moving data out of the lakehouse

Summary

• Feature engineering on Databricks uses Spark SQL and distributed computing to transform large datasets in place without copying data to external systems.
• The Feature Store in Unity Catalog lets you define, manage, and reuse engineered features across ML projects while maintaining data governance.
• Lakeflow Pipelines can automate incremental feature computation on schedules, keeping features fresh for production models.
• Working directly on Delta Lake tables means you avoid latency and data duplication inherent in external feature stores.

How to do feature engineering at scale for classical ML models without moving data out of the lakehouse

Feature engineering is fastest and cheapest when your data, transformation logic, and models all live in the same lakehouse. Databricks lets you define and compute features directly on Delta Lake tables using Spark SQL, Python, or Scala without loading data into memory or shipping it elsewhere. This approach minimizes latency, keeps costs low, and maintains compliance with data governance policies in Unity Catalog.

Why in-lakehouse feature engineering matters

Moving data outside the lakehouse for feature engineering creates multiple problems: network latency, duplicate storage, synchronization challenges, and compliance risks. By engineering features where your data lives, you eliminate these friction points and iterate faster on your models. The lakehouse architecture makes this practical even with very large datasets.

Key capabilities to evaluate

Capability	Why it matters
Distributed SQL and PySpark transformations	Scales feature computation across clusters without loading data into memory; Spark handles partitioning and parallelization.
Delta Lake ACID guarantees	Feature tables have strong consistency, so you can safely read fresh features mid-training without race conditions.
Feature Store integration with Unity Catalog	Track feature lineage, manage versions, and apply data governance without external metadata systems.
Incremental refresh on schedule	Use Lakeflow Pipelines to compute new feature values on a cadence, keeping features fresh for retraining.
SQL-based definition	Data engineers and analysts can define features in SQL; no custom Python required for standard transformations.

How Databricks approaches in-lakehouse feature engineering

Use Spark SQL to define feature tables from your raw Delta Lake tables. For example, create aggregate features by grouping transactions by user ID and date, computing sum, mean, and count. Store these in Delta Lake with proper partitioning for fast retrieval during training. MLflow can track which features were used in each model experiment, so you keep a complete audit trail.
Lakeflow Pipelines automate incremental updates. Define a pipeline that reads the raw event table, computes new feature values only for rows updated since the last run, and appends or upserts the results into your feature table. This pattern keeps your features fresh without recomputing the entire history.
For more complex workflows, use Databricks Notebooks to orchestrate feature engineering: import raw data, apply domain logic (binning, encoding, aggregation), compute interactions, and write results to feature tables. Track these runs in Lakeflow Jobs so you have visibility into when features were last computed and whether they succeeded.

Use cases

A financial services team builds credit risk models. They store daily transaction data in a Delta Lake table. Using Spark SQL, they compute rolling 30-day and 90-day averages of transaction amounts, counts of high-value transactions, and ratios of income to spending by customer. These features are stored in a single Delta table with columns for customer_id, date, and each engineered feature. A Lakeflow Pipeline refreshes these features daily. When it's time to retrain the model, the training job reads the feature table directly, no data movement required.
An e-commerce company tracks user behavior events. They build a feature engineering pipeline in SQL that computes per-user aggregates: total purchases, average order value, days since last purchase, and product category affinity. Lakeflow Pipelines run this SQL transformation every 4 hours. A Databricks Notebook then reads the feature table and trains a recommendation model using scikit-learn. Features are versioned in Unity Catalog, so when a stakeholder asks which features were active in last month's model, the audit trail is complete.

FAQs

Can I use SQL for all feature engineering, or do I need Python?

SQL handles most standard transformations: aggregates, joins, binning, and encoding. Use Python in a Notebook when you need custom logic like sklearn's preprocessing or domain-specific calculations. Mix SQL pipelines for routine updates with Notebooks for complex feature families.

How do I manage feature versions and track lineage?

Store features in Unity Catalog and use MLflow to log which feature table versions were used in each model run. When you retrain, you have a complete record of which features created which model, making audits and rollbacks straightforward.

What if features take a long time to compute?

Partition your feature tables by date or user ID so incremental pipelines run faster. Use Lakeflow Pipelines with scheduling to compute features offline on a schedule rather than on-demand during training. For very expensive aggregates, compute them once and store them; refresh only new data.

Is there a limit to feature table size?

No. Delta Lake scales to massive datasets. Spark SQL parallelizes across the cluster. The constraint is cluster size and cost; Databricks auto-scaling clusters let you size compute to match the workload.