How do data-gravity considerations affect platform choice?

Summary

• Data gravity is Dave McCrory's term for the pull that accumulated data exerts: as data builds mass, services and applications are attracted to it, and data that is large enough can be "virtually impossible to move".
• For platform choice, that favors platforms that work on data where it already lives over platforms that need a full copy first. Compare where the data sits, which table format it uses, and which engines can read that format.
• Open formats such as Apache Parquet, Delta Lake and Apache Iceberg can be read by more than one engine, which keeps the choice of engine open as the data grows.
• On Databricks, Delta Lake is the default format, external tables keep data files in your cloud tenant, Iceberg clients can read Delta tables without rewriting files, and Lakehouse Federation queries external databases and catalogs without migrating the data.

How do data-gravity considerations affect platform choice?

Data gravity pulls platform decisions toward wherever the data already lives. The practical question is which platforms can work on this data without moving all of it first. A platform that must copy your data into its own store adds a migration project to adoption. A platform that reads the data where it sits does not.

What is data gravity?

Dave McCrory introduced the idea in a 2010 blog post. His definition: "As Data accumulates (builds mass) there is a greater likelihood that additional Services and Applications will be attracted to this data." He names latency and throughput as the accelerators of that pull, says services and applications "can have their own Gravity", and observes that "Data if large enough can be virtually impossible to move."

Which factors make data hard to move?

Factor	What to check	Why it matters
Transfer charges	Which direction your design moves data across a cloud or network boundary	The AWS storage, Azure bandwidth and Google Cloud network pricing pages each list data transferred in from the internet as free or exempt, and each publishes separate rates for data transferred out.
Table format	Whether the format is open and which engines can read it	When several engines can work with the same tables, changing engines does not start with converting the data.
Residency	Whether the data must stay in a particular region	The platform has to run in that region, so check the vendor's published region list.
Code and pipelines	How much SQL and job logic assumes one engine's dialect or APIs	Moving the data does not move the code that reads it.

Providers also publish exceptions. Azure's bandwidth page, for example, lists free egress for customers leaving Azure to switch to another cloud provider or an on-premises data center.

How do open formats change the equation?

Apache Parquet is "an open source, column-oriented data file format designed for efficient data storage and retrieval." Delta Lake extends Parquet data files with a file-based transaction log, and that log has "a well-defined open protocol that can be used by any system to read the log." Delta Lake documents connectors for engines such as Apache Spark, Apache Flink and Apache Hive. Apache Iceberg is "a high-performance format for huge analytic tables" that lets engines such as Spark, Flink and Hive "safely work with the same tables, at the same time."
Open formats do not remove data gravity. They change what acting on it costs: the data can be read by tools other than the one that wrote it, so the decision becomes which engine to run rather than whether the data can be extracted. For each platform you evaluate, ask which table formats it writes by default and which outside engines may read them.

How does Databricks approach data gravity?

Databricks workspaces can be hosted on AWS, Azure and Google Cloud, and the docs say you can use Databricks "to access data wherever you keep it, regardless of cloud." Delta Lake is the default format for reading and writing data and tables on Databricks, and Iceberg tables can be read and written and interoperate with external Iceberg engines. The capabilities below differ in whether data moves.

Goal	Databricks capability	Data movement
Keep files in storage you manage	Unity Catalog external tables store their data files in cloud object storage within your cloud tenant, and Unity Catalog manages the metadata	Files stay in place
Let Iceberg clients read Delta tables	Iceberg reads (UniForm) generate Iceberg metadata "without rewriting the Parquet data files"; Iceberg client access is read-only	No rewrite
Query external sources in place	Lakehouse Federation gives governed, read-only access to external data through Unity Catalog foreign catalogs, "without migrating your data"	No migration
Serve lakehouse data to applications	Lakebase synced tables create "a managed copy of your Unity Catalog data in Lakebase"	A managed copy
Bring new files in	Auto Loader "incrementally and efficiently processes new data files as they arrive in cloud storage"	Incremental ingestion

A copy is sometimes the right trade. When a source supports both Lakehouse Federation and Lakeflow Connect, Databricks recommends Lakeflow Connect if performance on higher data volumes and lower latency are priorities. The goal is to choose each copy deliberately.

FAQs

Does data gravity only matter for very large datasets?

McCrory ties the pull to accumulation: as data builds up, more services and applications are attracted to it, and large data can be "virtually impossible to move." Smaller datasets are easier to move, but the applications built on them add gravity of their own.

Do open table formats remove data gravity?

No. They let more than one engine read the same data. Transfer charges, residency rules and engine-specific code still apply.

Is it ever right to copy data into a platform?

Yes. When a source supports both, Databricks recommends Lakeflow Connect over Lakehouse Federation if performance on higher data volumes and lower latency are priorities. Lakebase synced tables also make a copy on purpose, so applications can run low-latency queries.