Building With Community Connectors: Custom Data Sources, Managed Reliability

Summary

Lakeflow Connect provides 100+ managed connectors for SaaS applications, databases, and file sources, but organizations always encounter niche proprietary APIs or specialized SaaS tools not covered by managed connectors—a gap that Community Connectors address with open-source, AI-powered templates.
Community Connectors are built on the Lakeflow Connect foundation and inherit its built-in features—out-of-the-box observability, SCD Type 2 handling, automatic retries, and Unity Catalog governance for credentials, pipelines, data, and lineage—while allowing full customization for unique source requirements.
An AI skill can build a new Community Connector from API documentation in minutes, and the Lakeflow Connect interface supports five methods for full CDC ingestion, enabling teams to deploy custom connectors through the same UI and API used for managed connectors.

Building With Community Connectors: Custom Data Sources, Managed Reliability

Are you tired of building fragile, costly API scripts for niche data sources that lack enterprise-grade features like lineage, retries and unified governance? Community Connectors are built on the foundational power of Lakeflow Connect and bridge the gap between "DIY" and “fully managed." These new AI-powered, open-source templates help you build, manage and host your own ingestion paths while inheriting many of the robust, built-in features of the Databricks Platform. In this lightning talk, we’ll explore: Built-in tooling: Use out-of-the-box observability, SCD Type 2 handling, automatic retries and more Flexible customization: Use open-source templates to meet requirements that go beyond standard managed connectors Unified governance: Centrally manage credentials, pipelines, data and lineage via Unity Catalog Join us to get started and learn how they offer the flexibility of a DIY approach with the scale and reliability of a managed Databricks service.
Talk By: Peter Pogorski, Staff Product Manager, Databricks ;

Chapters

FAQs

What is the difference between managed connectors and Community Connectors in Lakeflow Connect?

Managed connectors are fully maintained by Databricks and cover 100+ popular SaaS applications, databases like SQL Server, MySQL, Oracle, and Postgres, and file sources like SharePoint and Google Drive. Community Connectors are open-source templates for sources not covered by managed connectors—such as proprietary internal APIs or niche SaaS applications—that teams can customize while inheriting many of the same built-in features.

What built-in features do Community Connectors inherit from Lakeflow Connect?

Community Connectors inherit out-of-the-box observability, SCD Type 2 handling for slowly changing dimensions, automatic retries, governed secrets management, and Unity Catalog integration for centralized lineage and access control. This gives custom-built connectors enterprise-grade reliability without requiring teams to build these features from scratch.

How does the AI skill build a Community Connector from API documentation?

The AI skill reads the API documentation for the target source and automatically generates a Community Connector template in minutes, reducing the time from needing a connector to having a working deployment. This video demonstrates the flow using the Shopify connector as an example, showing the build process from the add data page in the Databricks UI.

Why is a governed semantic layer for AI agents a key use case for Lakeflow Connect?

By turning data from all sources and silos into centralized, governed Unity Catalog tables, Lakeflow Connect creates a semantic layer that AI agents can query to answer critical business questions. This video explains that having all data in one governed, incrementally updated location is especially important when building AI agents that need reliable, up-to-date enterprise context.

Full transcript

[00:08] I'm Peter Pagorski. I'm a product manager on the LakeFlow Connect team at Data Bricks. And today I'm going to be giving a quick talk on our new community connectors. So I'm going to go over what community connectors are, how you can
[00:20] actually build one of them, and then how you can deploy a community connector to bring your most important data into the lakehouse. And I'm just going to start off with a quick bit of housekeeping. Community
[00:33] connectors are currently in beta and we have active development ongoing. So what I'm talking about in this session is what's current, but it's also subject to change.
[00:45] And first off, what is LakeFlow connect? So lakeflow connect is the built-in way to get data into data bricks. And we have over a 100 connectors which are already available. So you can deploy
[00:58] these connectors either via the UI or the API and bring your data into Unity catalog as govern tables and LakeFlow connect will incrementally read from the source to keep your lakehouse up to
[01:11] date. So we're only going to pull in the records that have changed each time a pipeline runs. And this enables you to turn all of your different data data silos into a centralized govern layer.
[01:25] And this is super important, especially when you're building a semantic layer for your AI agents. So you can use that data from all of your different data sources and silos to answer the most critical questions for your
[01:38] organization. And the coverage of Lake Connect is extremely broad. So we already have over 100 sources and this includes a variety of SAS applications. We have a lot of
[01:51] database sources like SQL server, MySQL, Oracle and Postgress. And then we also have a bunch of file sources like SharePoint and Google Drive. If you want to bring in your documents like Excel,
[02:04] CSV files, PDFs, basically any document you keep in SharePoint or Google Drive. And we continue to expand the breadth of our connectors. So, we're adding new connectors every month at a break neck
[02:18] speed. Like Ally mentioned in the keynote this morning, we've really accelerated our ability to deliver connectors quickly. So, we're continue continuing to expand the breadth here.
[02:31] However, in reality, we know your team's always going to have an additional source that's probably not covered by our managed connectors. You're probably going to have some proprietary internal API or you're gonna have a niche SAS
[02:45] application that's not actually covered under the manage connectors. And for these sources, you're going to want to think about building a custom solution for ingestion. But that also means you need to think about a lot of different
[02:58] challenges. You're going to have to think about things like how do you manage your authentication and secret management? What's going to happen when your schema evolves? How do you actually bring that schema evolution into your lakehouse? And then it's always
[03:11] extremely challenging to actually build out sufficient observability. So if something does go wrong on one of these pipelines, how can you quickly and easily discover what's happening? And I've spoken to a lot of customers
[03:24] where they have built one of these custom ingestion solutions, but ultimately they tell me that they miss the manage the management experience of Blake Flow Connect, the governance and the observability they get with these
[03:37] managed pipelines and this is why we built community connectors. So, community connectors are open- source connectors with templates that we provide and we have AI enabled
[03:50] skills that help you quickly and easily build a connector from almost any source. Um, and they have the same unified management and deployment experience of a manage lakeflow connect
[04:02] connector. So, let me dive into the important things about a community connector. There's really four things you want to know about here. The first is how you actually build these connectors. And
[04:15] behind the scenes, we're running on top of Spark's Python data source API. And there's a simple LakeFlow connect interface that gets implemented. So that LakeFlow connect interface is implemented as part of our AI enabled
[04:29] skills. And then second, we have governance. Your credentials will live as secrets in Unity catalog. So you don't have to worry about keeping like plain text credentials in a notebook where they're completely exposed. Anyone
[04:43] can get access. And then third is observability. And this is the really nice part about community connectors. You don't have to learn anything new here. It has the exact same um experience as any other link flow
[04:56] pipeline where you have the exact same UI, the same event log, um the same metrics and the same automatic retries that happen on the pipeline. And then finally is deployment where you can
[05:08] deploy these right from the add data page in your datab bricks workspace. So they're right there available in the UI and anyone within your organization can quickly and easily get started after you
[05:21] build one of these connectors. And there are two different ways of actually using a community connector. You can either use an existing connector or you could build your own. And if you're using an existing connector, we
[05:35] already have about 20 connectors that are open source and available in the data bricks lab repo. You can also deploy these ones directly from the ad data page. And if you want to, because they're open source, you can fork them.
[05:48] You can modify anything about the connector behavior and then redeploy them yourself. And you can still do that through the add data page. And then if a connector doesn't exist yet, you can
[06:00] build your own custom connector using the AI embedded skills I was mentioning. So as part of this workflow, you would specify with source documentation. You could basically input like any REST API
[06:13] doc, some swagger documentation, and the workflow is going to generate research and actually validate the connector for you. The workflow is primarily optimized for REST API sources, but you can use it
[06:26] for basically any type of source. And I'm going to start off by quickly demoing uh the first path, which is deploying an existing connector. And for this one, I'm going to showcase the
[06:38] Shopify connector. I did a pre-record this demo just to make the deployment quicker, but this is an actual community connector that you can use.
[06:51] So to start off, I'm in the open source repo where community connectors live. And this is part of data bricks labs. And within the repo, there are about 20 connectors that are already available. So you can choose any of these. And the
[07:04] one we're going to be deploying is Shopify. And as part of the AI generation of the community connector, it's going to generate a readme file. And this contains all of the information about the connector like what
[07:16] authentication methods are we using, what are the objects that are supported from the source and are there any features that have been implemented in the connector like supporting std type 2 history on ingestion. I can then go into
[07:30] the shopify.py file and this is where the actual implementation of the connector lives. It implements the lakeflow connect interface which I was mentioning and it utilizes the Python data source API
[07:43] and the implementation is going to incrementally ingest data whenever possible. So instead of doing a batch load of the object you're going to get the fastest and most efficient um
[07:55] ingestion experience possible. I can then go to the add data page and here I can see all the existing lakeflow connect manage connectors and then I can scroll down and we'll see the community connector section
[08:09] and all the tiles shown here are connectors that are open source and available today and for a connector that doesn't actually exist in that data bricks lab repo we also have a custom connector button in here you can specify
[08:24] any source you want to ingest from as well as a get repo link to the implementation of the connector that you've generated. So now for this demo, I'm going to click
[08:36] on the existing Shopify tile. And when I get into the connector setup page, I'm first going to choose a connection to the source. And when I create a new connection here, it's going to have all of the required
[08:49] properties that Shopify actually needs for me to authenticate to the source. And these properties are going to be determined when I'm generating the connector using our AI provided skills. And these credentials are also going to
[09:02] be governed under Unity catalog. So you don't have to keep your plain text credentials sitting around in a notebook. I'm then going to give my pipeline a name. I'm going to call it Shopify ingestion pipeline.
[09:15] Then I'll pick an event log location like any other pipeline. And this is where the events and logs from the pipeline will actually be stored. And then finally, I need to pick a root path folder. This root path folder is where
[09:28] all of the implementation and configuration files related to the connector will be stored. So just like any other pipeline, this is where those um artifacts exist.
[09:41] And then from here I land within the leak flow pipeline editor. And today we have a configuration file that you would update. But in the near future, we'll also enable you to select lists directly in the UI. And this is where I specify
[09:55] what objects I want to ingest from the source. So I'll specify the source tables, any target tables, what the destination catalog and schema is. And then I can enable any features I want on the connector like std type 2 ingestion
[10:09] just like I did here. And with this config file filled out, I can then run my pipeline and we'll actually start ingesting the data from the source. We can start to see the data land from Shopify and it's now available
[10:23] within my catalog. So just like that, we ingested data using a community connector. Now I'm going to walk through how you would actually build your own connector
[10:35] and how all of the different open- source connectors we have available got built in the first place. So step one of building a community connector, it's just going to be running a single skill command with the API docs
[10:49] for the source. You don't actually ever really need to read the API docs for the source yourself in order to successfully build a connector. You can basically just throw them into the scale and 90%
[11:02] of the time on the first try, we have a very successful connector implementation. So the skill is going to read the API interface and it's going to work out everything you need such as the authentication uh which endpoints matter any
[11:16] pageionation schema for the source and then how we're actually going to read incrementally so that we're not repulling all of your data on every single pipeline run and connector code can be some of the most repetitive code
[11:28] in data engineering. So you have pageionations, you have cursors, you have the same patterns over and over again. And this is why we found these AI enabled skills that we're able to build
[11:40] are so effective in building these connectors. The API docs are effectively the spec for the connector. The interface we've provided is narrow enough with the AI can fill it out and we confidently get some strong connector
[11:54] logic. On step two, the connector is actually implemented. So this consists of implementing four to five different methods in the link flow connect interface and these are using the Python
[12:07] data source API. Shared libraries are going to handle Spark the offsets and all of the pipeline retries. So when the AI is generating a connector, what it's actually doing is
[12:19] filling in these methods on the Lakeflow connect interface. Four of them are required and one of them's optional. So let me just quickly walk you through each of these methods. The first is list tables which is the connector telling
[12:32] the platform what's available. It's going to name every table which has been implemented in the connector from the source. So in the case of Shopify, this will be things like customers, orders, and products. Get table schema is then
[12:46] going to describe each of those tables, the columns, their data types. So the platform knows what the shape of the data is when it's coming in. And then we're going to have read table metadata.
[12:58] And this is where it's doing the hard work. It's actually going to be pulling the data uh which is pageionated through the source and checkpointed. So a failure halfway through means you don't
[13:10] have to restart your pipeline and reingest all of that data. And then finally, retable deletes is optional. you only need to implement it if your source actually captures uh deletes and
[13:23] if you want to have that true CDC experience. So that's the entire linkflow connect interface. Pretty straightforward to implement. And then on step three, it's not just going to stop at generating the code.
[13:36] Our workflow will actually test and validate the connector for you. So first it's going to test and validate against a simulated version of the API. You don't actually even need credentials for the source to get started. And then it's
[13:50] going to run it against the live API where it's going to see are there any issues that I need to iterate on and fix. And the workflow will go back and actually improve on the connector.
[14:02] And then at this point, you've successfully built built a connector. So the connector is going to be validated. It's ready to deploy in your own workspace and it can be run as a typical governlow pipeline. You could also open
[14:16] a pull request. So we accept requests from the community if you want to share your connector so that any other datab bricks uh customer could actually deploy that connector you just built. And I'm just going to quickly recap what
[14:29] I went through in the presentation. So what can you build a community connector for? It is optimized for REST API sources, but it supports almost any type of source. And our goal is just to make
[14:41] it easy to build any connector in just a few hours. They run as native LakeFlow pipelines with built-in observability and unity catalog governance permissions and lineage just like any other Lakeflow
[14:53] connect manage connector. And they're priced using standard LakeFlow pipeline pricing. So they run on standard serverless or classic compute clusters. You're just playing paying for the
[15:05] compute. you're not paying for any additional data extraction fees. So everything I showed you today is already open source and available in datab bricks labs and you can check it
[15:18] out uh by navigating to this GitHub link. So you can deploy a connector from the ad data page or get started and build your own and we're very excited to see what everyone builds. So with that I'll hang around afterwards and answer
[15:31] any questions you have but thank you for joining.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.