How can I get my app telemetry into Databricks?

Summary

• The most direct way to get app telemetry into Databricks is Zerobus Ingest, a push-based streaming API in Lakeflow Connect that writes records straight into Unity Catalog Delta tables with no message bus to run.
• Services that already emit OpenTelemetry can send traces, logs and metrics to the native OTLP endpoint in Zerobus Ingest.
• If telemetry already flows through Apache Kafka, Structured Streaming and Lakeflow pipelines read the topics directly. If it lands as files in cloud storage, Auto Loader picks up new files as they arrive.
• After the data lands, Lakeflow pipelines validate it with expectations, and Databricks SQL alerts notify you when a query condition is met.

How can I get my app telemetry into Databricks?

Push it straight into a table. The shortest path is usually Zerobus Ingest, which the Databricks docs describe as "a push-based streaming API that writes data straight into Unity Catalog Delta tables at high scale, with no message bus to run." Services that already speak OpenTelemetry can point an exporter at its OTLP endpoint. If your telemetry already sits in Apache Kafka or in files in cloud storage, read it from there with Structured Streaming, Lakeflow pipelines or Auto Loader.

What counts as app telemetry?

App telemetry usually comes in four kinds:

• Logs: timestamped records with a severity and a message body.
• Traces: spans that follow one request across services, with timing, status and parent-child links.
• Metrics: numeric measurements such as request counts, error counts and latency histograms.
• Events: discrete product or business actions, such as clicks, sign-ups and state changes.

Traces, logs and metrics are the three OpenTelemetry signals. The Zerobus Ingest docs list observability data ("OpenTelemetry traces, logs, and metrics") and application and clickstream events among its documented use cases.

Which ingestion pattern fits your telemetry?

Where the telemetry is today	Databricks path	How it works
Emitted by OpenTelemetry SDKs or the OpenTelemetry Collector	Zerobus Ingest OTLP endpoint	Exporters send traces, logs and metrics over OTLP/gRPC or OTLP/HTTP (Protobuf) into tables you create in advance
Produced by your own services or devices	Zerobus Ingest SDKs or REST	Producers push JSON, Protocol Buffers or Apache Arrow records into a managed Delta table or a streaming table
Already flowing through Apache Kafka	Structured Streaming or Lakeflow pipelines	A streaming read with the kafka source, or the read_kafka SQL function in streaming tables and Lakeflow pipelines
Written as files to cloud storage	Auto Loader	The cloudFiles source processes new JSON, CSV, Parquet, Avro and other files incrementally as they arrive

How does Zerobus Ingest work?

Setup follows four documented steps: get a Zerobus Ingest URL, create or identify the target table, create a service principal and grant it privileges on that table, then connect a client or exporter. Producers can use SDKs over gRPC (Python, Java, Rust, Go and TypeScript), which the docs call best for high-volume streaming producers, or REST, best for large fleets of lightweight or chatty edge devices. The Zerobus Ingest release stages page lists gRPC, REST and OpenTelemetry (OTLP) as Generally Available.
The docs say records land in Delta within seconds. Delivery is at-least-once "when the protocol provides a durability acknowledgment and the client retries records that aren't acknowledged." A retried record can arrive twice, so plan for deduplication. Zerobus Ingest is only available in some regions, so check the ingestion availability list before you start.

How do you send OpenTelemetry data?

Zerobus Ingest OTLP is "a native OpenTelemetry Protocol (OTLP) endpoint built into the Zerobus Ingest service." Any OTLP-compatible exporter, such as an OpenTelemetry SDK or the OpenTelemetry Collector, can send to it. The rules:

• Encodings are OTLP/gRPC and OTLP/HTTP with Protobuf. Requests with a Content-Type of application/json are rejected.
• Tables must exist before data arrives, using the documented schemas for spans, logs and metrics. Zerobus Ingest does not create or modify tables.
• Each request names its target table in the x-databricks-zerobus-table-name header, so traces, logs and metrics each get their own exporter pipeline.
• The service principal needs USE CATALOG, USE SCHEMA, and MODIFY and SELECT on each table. In the Collector, the docs use the otlp exporter for gRPC and the otlphttp exporter for HTTP.

What do you do with telemetry after it lands?

Lakeflow pipelines can read the raw tables into streaming tables, where each record is processed exactly once assuming an append-only source, and build materialized views for rollups. Expectations validate records as they flow: keep and count bad rows with EXPECT, drop them with ON VIOLATION DROP ROW, or stop the update with ON VIOLATION FAIL UPDATE. Data quality metrics are in the pipeline event log.
For table layout, Databricks recommends liquid clustering for all new tables, including streaming tables and materialized views. It replaces table partitioning and ZORDER, and CLUSTER BY AUTO lets Databricks choose the clustering keys. To act on the data, Databricks SQL alerts run a query on a schedule and notify you when a condition you define is met, such as an error count crossing a threshold.

FAQs

Can I use OpenTelemetry with Databricks?

Yes. Point an OTLP exporter at the Zerobus Ingest OTLP endpoint over OTLP/gRPC or OTLP/HTTP with Protobuf. JSON-encoded OTLP requests are rejected, so set the exporter protocol to one of the supported encodings.

Do I need Kafka to stream telemetry into Databricks?

No. Zerobus Ingest writes into Delta tables with no message bus to run. If you already run Kafka, Structured Streaming can use it as a source or a sink.

Does the OTLP endpoint create tables for me?

No. Create the spans, logs and metrics tables with the documented schemas first, and grant the service principal access before you start sending.

What if my telemetry already lands as files?

Use Auto Loader. It reads from Amazon S3, Azure Data Lake Storage, Google Cloud Storage and Unity Catalog volumes, and tracks processed files in its checkpoint so each file is processed exactly once. Databricks recommends Auto Loader in Lakeflow pipelines for incremental ingestion.