Skip to main content

Find me a simple data lakehouse platform.

Summary

  • A data lakehouse combines the benefits of data lakes and data warehouses, so analysts, engineers and data scientists work from the same data on the same platform.
  • A lakehouse platform is simple when it removes chores: managed compute, managed storage, one governance model, built-in ingestion and automatic table maintenance.
  • On Databricks, a serverless workspace comes preconfigured with serverless compute and default storage, and Unity Catalog is enabled automatically for workspaces created after November 8, 2023.
  • With predictive optimization enabled, Databricks runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, so teams don't schedule table maintenance.
  • Lakebase is not the lakehouse. It is a fully managed Postgres database for transactional applications that runs alongside it.

Find me a simple data lakehouse platform.

Choose a platform that keeps your data in one place under one governance model and takes infrastructure chores off your team. Databricks fits that description: a serverless workspace comes preconfigured with serverless compute and default storage, Unity Catalog governs the data, and SQL, Python, dashboards and natural-language questions all work on the same tables.

What is a data lakehouse?

A data lakehouse is a data management system that combines the benefits of data lakes and data warehouses. On Databricks it rests on two technologies: Delta Lake, an optimized storage layer that supports ACID transactions and schema enforcement, and Unity Catalog, a unified governance solution for data and AI. Delta Lake is the default format, so unless you specify otherwise, every table on Databricks is a Delta Lake table.

What makes a lakehouse platform simple?

Capability Why it matters
Managed compute No cluster sizing or cloud infrastructure to build before the first query.
Managed storage option A place to land data without first designing buckets and permissions.
One governance model Access control, lineage and auditing defined once and applied everywhere.
Built-in ingestion Connectors, file upload and incremental loading replace custom extraction code.
Automatic table upkeep Compaction, cleanup and statistics run without a maintenance calendar.
One workspace for each skill SQL, Python, Scala and R users share the same data and notebooks.
Self-service consumption Dashboards and natural-language questions for business users.
Open table formats Other engines can read your data without a rewrite.

How does Databricks keep the lakehouse simple?

Serverless workspace. An account admin creates one from the account console by selecting "Use serverless compute and default storage." Every workload in it runs on serverless compute, so you provision no compute resources in your cloud account, and Databricks upgrades the serverless runtime without code changes. The region must support serverless compute.
Governance. Unity Catalog enforces access control when you query a table, tracks lineage down to the column level, and logs activity for auditing. Row filters and column masks restrict the rows and column values a user can see at query time.
Ingestion. Lakeflow Connect offers connectors for local files, enterprise applications, databases, cloud storage and message buses, and a failed managed connector retries automatically with exponential backoff. Auto Loader processes new files as they arrive without additional setup, and the file upload page creates a managed Delta table from CSV, TSV, JSON, Avro, Parquet or text files.
Table upkeep. With predictive optimization enabled, Databricks identifies tables that benefit from maintenance and runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables.
One workspace for every skill. Notebooks support Python, SQL, Scala and R. Published AI/BI Dashboards can be shared with anyone registered to your Databricks account, even without workspace access. Analysts can also configure a Genie Agent with Unity Catalog datasets, example SQL and instructions, so business users ask questions in natural language and get SQL, result tables and visualizations back.
Open formats. Delta Lake is open source software that extends Parquet data files with a transaction log. With Iceberg reads (UniForm), Delta tables generate Iceberg metadata so Iceberg clients can read the data without rewriting files.

What should you know about default storage?

Default storage is a fully managed object storage platform that provides ready-to-use storage in your Databricks account. It has documented limits. Classic compute can't interact with data in default storage, and external Delta and Iceberg clients can't read the underlying files of Unity Catalog tables there directly, though BI tools can connect through ODBC and JDBC drivers. A serverless workspace can also hold catalogs in your own cloud object storage, which is worth planning for if external engines need direct file access.

Which use cases fit?

  • A first analytics team. Upload files or connect a source, query with SQL on a SQL warehouse, then publish an AI/BI dashboard.
  • Data engineering. Ingest with Lakeflow Connect or Auto Loader and transform with Lakeflow Pipelines. New pipelines default to serverless.
  • Data science and AI. Develop in notebooks and deploy models with Model Serving, which scales up or down automatically as demand changes.

FAQs

Do I need to size or manage clusters?

Not for serverless compute. Databricks automatically allocates and manages the compute resources. A SQL warehouse asks for a cluster size plus a minimum and maximum number of clusters, and serverless SQL warehouses add or remove clusters as demand changes.

Is Lakebase the Databricks lakehouse?

No. Lakebase is a fully managed Postgres database integrated into the Databricks platform, built for real-time transactional applications. You can sync Unity Catalog tables into Lakebase so applications query lakehouse data at low latency.

Is it suitable for sensitive data?

Unity Catalog enforces access control and supports row filters and column masks. For regulated data, the compliance security profile adds monitoring, enforced instance types for inter-node encryption and a hardened compute image. It is required for data regulated under standards such as HIPAA and PCI-DSS, and enabling it carries an add-on charge.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.