How do I build my own ingestion connectors if one doesn't exist on Databricks?

Summary

• Check managed and community connectors first. Lakeflow Connect covers database, SaaS, file and streaming sources, and community connectors extend it to sources without a managed option. If none supports your source, you can build a custom connector and run it in your workspace.
• A custom connector is a Python class that implements the LakeflowConnect interface: list_tables, get_table_schema, read_table_metadata, read_table and, optionally, read_table_deletes. It wraps the Spark Python Data Source API and runs as an ingestion pipeline backed by Lakeflow Pipelines.
• The flow is implement, pass the required generic test suite, flatten to a single-file artifact, package dependencies as wheels in a Unity Catalog volume, then deploy from the Add or upload data page or with community-connector publish.
• Databricks documents custom connectors as Beta, and connectors contributed to the community repository aren't backed by Databricks SLAs. Files, message buses and application writes have native ingestion paths that need no connector.

How do I build my own ingestion connectors if one doesn't exist on Databricks?

Build a custom connector with the LakeflowConnect interface from the Lakeflow Community Connectors repository on GitHub, then deploy it in your own workspace as an ingestion pipeline. You write the source-specific parts in Python: authentication, table discovery, schemas and incremental reads. Shared framework libraries handle the Spark integration, streaming and offset management.
Lakeflow Connect organizes connectors by source type, with community and custom connectors for sources that have no built-in option. Managed connectors produce pipelines governed by Unity Catalog and powered by serverless compute and Lakeflow Pipelines. Community connectors are built and maintained by the community. The custom connector feature is in Beta.

Which ingestion route fits your source?

Your situation	Route
A database, SaaS application, file store or message bus that Databricks supports	Managed connector
A community-built connector already exists	Community connector (Beta)
No managed or community connector exists	Custom connector on LakeflowConnect (Beta)
You need custom partitioning or parsing	The Python Data Source API directly
Files land in cloud object storage	Auto Loader (cloudFiles)
Your own application can send the records	Zerobus Ingest, with an SDK over gRPC or the REST API
Events arrive on Kafka, Kinesis or Pub/Sub	A Structured Streaming source

How do you build a custom connector?

1. Set up. You need Python 3.10 or above, a workspace with Unity Catalog enabled, API credentials for the source and Git. Clone databrickslabs/lakeflow-community-connectors and create your connector under sources/<your-source>.
2. Implement the interface. LakeflowConnect is an abstract base class:

Method	What it does
list_tables()	Returns the names of the tables the source supports
get_table_schema(table_name, table_options)	Returns a Spark StructType
read_table_metadata(table_name, table_options)	Returns primary_keys, cursor_field and ingestion_type
read_table(table_name, start_offset, table_options)	Returns JSON-compatible dict records and the next offset
read_table_deletes(table_name, start_offset, table_options)	Optional, needed only for cdc_with_deletes

The ingestion_type is snapshot, append, cdc or cdc_with_deletes. Snapshot tables are read in batch and the other three as streams. The framework calls read_table repeatedly to paginate and stops when the offset it gets back equals the offset it passed in.
3. Test. The generic test suite is required and write-back tests are recommended. Add unit tests for pagination, type coercion and error recovery.
4. Package and publish. A connector deploys as a single-file source artifact plus Python wheels for its dependencies, which must sit in a Unity Catalog volume.

python -m pytest tests/generic/ --connector <your-source> --credentials credentials.json
python tools/scripts/merge_python_source.py --connector <your-source>
community-connector publish <your-source> \
  --spec src/databricks/labs/community_connector/sources/<your-source>/connector_spec.yaml

To deploy in the UI instead, go to +New, Add or upload data, Community connectors, and add the custom connector. Then create the ingestion pipeline and edit ingest.py. Its options include the required connection_name and objects, plus destination_catalog, destination_schema, scd_type (SCD_TYPE_1 by default, SCD_TYPE_2 or APPEND_ONLY) and primary_keys.

What if LakeflowConnect doesn't fit?

Implement the Python Data Source API directly when you need greater flexibility, such as custom data partitioning or parsing into a specialized schema. The repository doesn't recommend it as the default. The API arrived in Spark 4.0, and on Databricks, PySpark custom data sources need Databricks Runtime 15.4 LTS and above or serverless environment version 2. You subclass DataSource, return a DataSourceReader for batch reads or a DataSourceStreamReader for streaming, register the class with spark.dataSource.register, then read it with spark.read.format (or spark.readStream.format for streams).

When does a custom connector make sense?

• Proprietary or internal services. The PySpark custom data sources docs note that Spark has no built-in support for many systems, including REST APIs and proprietary internal services.
• Sources that report deletions. Use cdc_with_deletes and implement read_table_deletes, and deleted records are applied as deletes to the destination table.

FAQs

Do I have to contribute my connector to the community repository?

No. A custom connector is deployed and run in your own workspace, with no community registration required. If you want others to use it, open a pull request. Contributed connectors become community connectors, which the community maintains and which aren't backed by Databricks SLAs.

What language do I write it in, and where do credentials go?

Python. LakeflowConnect is a Python abstract base class. The required connection_name option of the pipeline points to the connection that stores the source's authentication credentials.

Is a custom connector ready for production?

The docs mark the feature as Beta, and the community connector docs warn that interfaces and behavior are subject to change. Treat the connector as code you own: test it against the real source and plan for interface changes.