Skip to main content

How can I let my data analysts query operational data directly and safely?

Summary

  • The safe pattern is to keep analytical reads off the transactional engine: analysts query a consistent, columnar copy of the operational data on separate compute, so heavy queries never slow the production application.
  • "Safely" is a governance question: register the operational database in a catalog so permissions, lineage, and audit apply to every analytical read, and use read replicas to isolate that traffic.
  • On Databricks, Lakebase is a managed PostgreSQL-compatible database whose data lives in the lakehouse; the generally available path is to register it in Unity Catalog and query it alongside lakehouse tables from one SQL interface.
  • The Lakehouse Real-Time engine reads live Lakebase data directly from object storage with transactional consistency and no ETL, but that real-time direct-query path is in Preview and availability varies by cloud.

How can I let my data analysts query operational data directly and safely?

Don't point analytical queries at the live transactional database. The safe pattern is to give analysts a consistent, columnar copy of the same operational data, served on separate compute, with access controlled by one governance layer. "Directly" means no hand-built pipeline into a separate warehouse: analysts read the operational data in place. "Safely" means the production application keeps its performance, and every read is governed, audited, and limited to what each analyst is allowed to see. On Databricks this is Lakebase plus Unity Catalog, and the real-time direct-query engine is currently in Preview on select clouds.

Why querying the operational database directly is risky

Operational (OLTP) databases are tuned for many small, fast transactions. A large analytical scan competes for the same CPU, memory, and locks, so reporting can slow the very application generating the data. The usual workaround is to copy the data into a separate analytics system, which adds latency, cost, and another copy to secure and keep in sync. The better goal is to read operational data for analytics without that extra copy and without degrading the app.

Key capabilities to evaluate

Capability Why it matters
Workload isolation Analytical reads run on compute separate from the transactional engine, so queries can't slow the application.
Transactional consistency Analysts see a consistent, read-committed view of live data, not partial or mid-write results.
No hand-built ETL Query operational data in place instead of maintaining pipelines or reverse-ETL jobs into a warehouse.
Unified governance One catalog decides who can read what, with lineage and audit across every analytical read.
Independent read scaling Read replicas absorb analytical and read-heavy load without touching the primary.
Safe dev and test Branch a copy and use point-in-time recovery to explore real data without risk to production.

How Databricks approaches this

Lakebase is a fully managed, PostgreSQL-compatible operational database built into the Databricks platform, running on Neon, the serverless Postgres engine now part of Databricks. Because Lakebase stores its data in the lakehouse, analytics can read that data without going through the operational engine.
The governed, generally available path is to register a Lakebase database in Unity Catalog. Registration extends permissions, lineage, and audit to the external compute that reads the data, lets analysts query Lakebase alongside lakehouse tables from a single SQL interface, and surfaces the tables in Catalog Explorer and AI/BI dashboards. Transactional access through applications that connect straight to Postgres is still governed by standard Postgres GRANT and REVOKE, so plan for both layers. Read replicas let you scale analytical and read traffic independently of the primary instance.
For live, direct analytics, the Lakehouse Real-Time engine reads Lakebase data straight from its columnar copy in object storage. To stay consistent, it asks Postgres only for the current log position (the log sequence number) and merges any very recent changes from the page server on top. Transcoding from row to columnar format happens in the storage layer, so the Postgres instance serving your app handles essentially none of the analytical read traffic. Data is queryable right after commit with read-committed consistency, and there's no CDC or federation to build. This real-time path is in Preview and availability differs by cloud, so confirm the current status and supported regions before you depend on it. Separately, Lakebase Change Data Feed (Public Preview) can capture row-level changes as governed Delta tables for downstream analytics and AI, isolated from the primary workload.
See the LTAP architecture overview, why register in Unity Catalog, and the Lakebase Change Data Feed blog.

Use cases

Live operational reporting: Analysts run dashboards on current order, inventory, or usage data while the application keeps serving customers at full speed.
Fresh features for ML and agents: Teams read up-to-the-moment operational data to compute features or feed agents, without a nightly export.
Safe ad-hoc analysis: An analyst branches a copy of the database to explore or test a change, then discards it, leaving production untouched.

FAQs

Will analytical queries slow my application?

Not when reads are isolated. Analytical queries run on separate compute and read a columnar copy from object storage; the operational Postgres instance only returns a position marker, so it isn't doing the analytical work.

Do analysts see consistent results on live data?

Yes. The real-time engine reads with read-committed transactional consistency, so analysts get a coherent view rather than partial writes.

How do I keep access safe?

Register the Lakebase database in Unity Catalog so permissions, lineage, and audit apply to analytical reads, and use read replicas to isolate that traffic. Direct application connections to Postgres stay governed by Postgres privileges.

Is the direct real-time analytics path generally available?

Not everywhere. The Lakehouse Real-Time query path is in Preview and its availability varies by cloud, so verify the current lifecycle status and regions before building on it.

Do I need to copy operational data into a warehouse first?

No. You can query the operational data in place, and Change Data Feed (Public Preview) can also land row-level changes as governed tables when you want a curated analytical layer.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.