Skip to main content

Scalable Data Ingestion: Building Custom Pipelines with the Fivetran Connector SDK and Databricks

Summary

  • Fivetran Connector SDK: This video covers building custom data pipelines using the Fivetran Connector SDK integrated with Databricks on the Databricks Data and AI platform.
  • Scalable Data Ingestion: Learn how to architect scalable data ingestion pipelines for moving data from diverse sources into your lakehouse efficiently.
  • SDK Capabilities: Discover the tools and frameworks provided by Fivetran SDK for rapid connector development and deployment.

Scalable Data Ingestion: Building Custom Pipelines with the Fivetran Connector SDK and Databricks

Watch: Scalable Data Ingestion: Building Custom Pipelines with the Fivetran Connector SDK and Databricks
Talk by CL Abeel, Senior Solution Architect, Fivetran (Sponsor Speaker); Kelly Kohlleffel, Sr Global Director, Partner Sales Eng, Fivetran (Sponsor Speaker)
Organizations have hundreds of data sources, some of which are very niche or difficult to access. Incorporating this data into your lakehouse requires significant time and resources, hindering your ability to work on more value-add projects. Enter the Fivetran Connector SDK- a powerful new tool that enables your team to create custom pipelines for niche systems, custom APIs, and sources with specific data filtering requirements, seamlessly integrating with Databricks. During this video, Fivetran will demonstrate how to (1) Leverage the Connector SDK to build scalable connectors, enabling the ingestion of diverse data into Databricks (2) Gain flexibility and control over historical and incremental syncs, delete capture, state management, multithreading data extraction, and custom schemas (3) Utilize practical examples, code snippets, and architectural considerations to overcome data integration challenges and unlock the full potential of your Databricks environment.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is the Fivetran Connector SDK?

This video explains that the Fivetran Connector SDK enables developers to build custom data connectors for integrating data sources with the Databricks Data and AI platform.

How do you build custom pipelines with the SDK?

According to this video, the Fivetran SDK provides tools and frameworks for rapid connector development, enabling scalable data ingestion into Databricks.

What sources can you integrate?

This video demonstrates how custom pipelines built with the Fivetran SDK can connect diverse data sources to your Databricks lakehouse.

What are the benefits of SDK-based pipelines?

This video highlights how SDK-based pipelines enable rapid development, scalability, and reliability for data ingestion on the Databricks Data and AI platform.

Full transcript

So, we're not going to ask everybody to go around the room and introduce, but we would like to get your perspective. Does it Has everybody hit that uh QR code there with their phone? So, y'all are ready to go. I am going to jump out to a what? We have three questions right now. Should be really easy. The first one is I'm always interested in this one. I mean, we're at a data conference, but I, you know, Fiverr is not at the scale of data bricks in terms of size and all that kind of thing.
How familiar are you with Fiverr? Uh, and it's okay to say, "Yeah, I don't know. This just looked like a cool session to go to. I don't know anything about Fiverr." Yeah. So, here we go. A lot of lot of folks heard of it, maybe haven't used it before. A few people that are very familiar, maybe any Fiverr customers in the Okay. Yeah. Yeah. Okay. guys got something this side heavily weighted to fiverr customers. I don't know what's going on.
Okay, cool. All right, keep your keep your minty up. What do you think, CL? Any any surprises there so far? No, it's pretty evenly distributed. Heard of it, but never used it. About half. All right. All right. Let's go on to our next one. Now, I don't know if you're uh data leaders within your organization, your data engineers, application developers. The next question is, have you ever asked your team or have you ever been asked to build a custom connector?
And when I think custom connector, CL, I mean you're talking design, build, automate, test, deploy, refactor, maintain. It just kind of never ends. It absolutely never ends. So, let's see where we are. So, we've got a Okay, we've got some folks with experience in here. Absolutely. Building custom connectors. We're going to show you this is going to be CL is going to build a custom connector but it may be a little bit different than what you're thinking.
You know Fiverr has done a pretty good job of solving for instance ingesting data into data bricks all the merging and everything. We do that really well. We got thousand data bricks. Oh ramen in the room I better say this. We got the data bricks data integration partner of the year this year which we were really happy about. So uh yeah that was a that was a cool award. Really appreciate the data bricks team and all the work that we do together.
Okay. So, it's it's kind of a a little bit split here. Let's see what we got. Last one. First word that you think of when you think, "Oh, man. Somebody just asked me to build a custom pipeline." So, this could be yes. I'm really excited about this. It may be, "Hello, Claude. Are you there today?" I don't know what what your first words are, but uh let us know. I'm gonna I'm gonna ask you later on like what's your what's your process?
How did you do 40 or 50? I like that. Yeah. SDK. Okay. Yeah, it's tough. I when I ask this question, I get everything, oh man, not again, please. You know, keep me out of this thing. Yes. Ongoing maintenance is I think these bubble up the more people vote for them. They get a little bit bigger. So, cool. All right. Thank you'all. Gives us a little sense for uh for in the room kind of what we have. So, thank you all for that.
Any uh anything that's a surprise? I see a headache on there. Maintenance. Yeah, I was thinking details brittle. What's the first thing as a as a using the Fiverr custom connector SDK, the connector SDK, what's the first word that comes to mind for you? Uh, I think it's multiple words, but I think you only have to worry about the source. All right. So, maybe left left side, left side. And I ask you, not really that, but I asked you this morning.
So, you're really having fun building custom connectors. It's not it's not an E or Oh, man. This is terrible. Okay, thank you all very much. Let's go back. We've got uh got a lot of hands-on time today, but want to set a little bit of the stage. This room seemed pretty weighted over to Fiverr, maybe not as much over here. So, want to give you a little bit of a of an idea. So, automated data movement, scalable, reliable, secure, all the things that you need.
I think when uh you know, when I think of Fiverr, what I got asked for all the time was, make it simple. Uh don't don't make me deal with infrastructure. uh give me secure, give me self-s serve, all those kind of things. So that those can all be framed up technically, but ultimately we want to help deliver high quality, trusted, usable data into the data bricks data intelligence platform each and every time for any source over and over and over again to deliver any data bricks data workload and they are continuing to expand.
So se we talk about every source, every destination, every innovation. Anything that stands out to you on this slide? Every is a big word. Every Yeah, that is a very big word. How do we go? By the way, we've got we've got Does anybody know fiverr connectors in our connector library? So I started three years ago. We had about 150 175. Anybody want to let me let me see raise hands. You think we have over 500? Anybody over 500?
Okay. Anybody think we have over 600 in our library today that you can just pull out of the box and start using? Okay, we got a couple of 600. Anybody think we're over 700? Okay, we got a few of those. We actually are over 700. It's unbelievable. But CL, I know you could ask. You said you build 50 custom. CL, what about connector 701, 702, 703? That's what we're going to talk about today. By the way, every destination, you know, you're going to hear a lot, probably heard a lot in the keynote this morning. um support for iceberg, support for delta, like any data bricks configuration that you want to do within your organization.
A lot of our customers have multiple data bricks destinations. We will support that. A lot of cool ways to do it and certainly any any and every data bricks innovation. Talked about this. We're going to go into connector SDK databases, apps, file systems, event systems, anything that you want there. And some really cool stuff like I said with data bricks. I'm going to keep blowing through this because I want to get to your your buildout.
Um this is important though. So you go okay what is this connector SDK? When did we just launched this GA not too long ago? Yeah it was a couple months maybe. Yeah not not very long ago. So we've been working with it for a while uh internally and with partners and customers. But again kind of going back to what I talked about I I was uh doing some things last week and just going through this cycle you have to do to build end to end the uh open AI cool quote here on on what they're doing with us today.
So, I didn't I I want to kind of move quickly through that, but you know, you talk about somebody that probably has everybody used chat GBT at least once or twice. I don't want to see which like don't give me a claw, you know, let's not have a model war right now. Uh I like the way, by the way, data bricks just put in whatever you want. There's there's no more model wars. It's just we'll support whatever you have. This is what I want to get to though.
This is kind of like the meat of what we're going to talk about. So I talked about this design all the way to you know refactor and maintain you have to interact with a source with a connector I got to interact with Oracle SQL server whatever that source is in the I'll call it the middle part the middle box I need to orchestrate maybe I normalize maybe I'm getting my data types ready to go because at the end I've got to interact with data bricks I got to know how to do data bricks right and they're very particular about how we integrate with uh their platform.
We have to do it right. CL, we're taking out twothirds of this custom build process, aren't we? We absolutely are. We've already done optim all the optimization of writing to data bricks. We do the table creation on the data brick side. If there's any changes in the source, we do the alter table statements on the right hand side. All that's already been taken care of by the fivet platform. So, I'm kind of left with uh I see a connector. py there.
Uh you want to talk about that? Is is this kind of the key to the connector SDK? That is the key. So for connector SDK, the first step is installation standard pip install command. PIP install fiverrren connector SDK. And then we'll walk through the connector. py file, which is the one file that you have to have for your custom connector. You can have more. We support multiple Python files if you prefer to have that structure.
Uh but connector. py and the uh the update method inside connector. py is the entry point for your custom connector. And we'll show you more detail about that in a minute. So that little cloud source system via source API that becomes your kind of the custom piece and there's just no reason to build the other two/irds anymore. At least in our opinion you're you're causing additional complexity kind of going back to that word cloud and all that kind of stuff, right?
Um all right. So CL this is the flow today. want you want to kind of preview this for us what what you're going to do and how you're going to do it from a data source perspective on into data bricks. Sure. Yeah. So, we sort of had a scenario on our minds. Um you're a restaurant chain and you have 100 locations all throughout the US and you want to monitor the weather forecast for your location. So, you might want to send out a promotion.
If it's going to be rainy, maybe you get delivery. If it's going to be nice, come and dine on our patio. Uh so, we're going to hit the National Weather Service. Um, but since we're a restaurant, we don't have latitude longitude for all the zip codes where our locations are. So, we're going to hit a second API called Zippopotamus to get latitude longitude for our zip codes that we can send then send the National Weather Service, get the weather forecasts, uh, build that interconnector. py.
Uh, that gets upserted into Fiverr. Fiverr takes care of schema creation, table creation over on the data brick side. I'd rather deal with zip codes anytime rather than long. Yeah, you're a headache for me. I don't like dealing with lat longs. What do you think? Are you ready? Let's do it. Yeah, let's do it. All right, let's get over to I'm going to we're going to switch spots here. And by the way, you may have seen us up here.
Uh Seel's Windows. uh the AB was not liking uh Windows today. So, can we zoom in a little bit? I think it might be hard for the folks to read. Let us know. Yeah, thank you. Let us know when y'all can see that a little bit better. Let me rearrange here. Get down a little bit more. Is that okay for everybody? You want me to go one more? I think we need to go one more. Yeah, another. I can go another. Is that Is that okay for everybody?
Okay, let me go little screen here. So, scrambled a little bit at the end to um to do this. But, uh hopefully we should have everything we need here. CL, do you want to uh Yeah, let's just bring it up to connector. py. All right, great. I can thanks I can uh navigate around if you tell me what to do. All right, so we talked a little bit about sort of the use case that we had in our minds for to walk you through weather advanced.
Weather advanced. Great. Thank you very much. So CL had her own project set up on on her laptop. So bear with us just a just a little bit. So a lot of this here on the screen is is Python which you all if you're in this room probably good at doing and if you're not you have a AI friend that can help you out. So, we're not really going to Do we have a Java version? We don't at this point. We only support Python. Nothing but Python.
Nothing but Python. The reason I asked, I had a friend of mine that reached out to me. Hey, I want to build some custom con. You got a job? I'm a Java guy. No, we got Python though. He's built three custom connectors. Never never done anything in Python before. So, Python only. Python only for now. Uh, so the main question that you probably all have in your heads is like, so I'm good at writing Python. I can write a function that talks to this API, that API.
What does Fiverr need in order to write the data that I've queried from my source? What does Fiverr need to know to write that into data bricks? So like I mentioned earlier, the first step is pip install fiverrren connector SDK and then the required imports at the top of your Python code are connector logging and operations. So the connector u this uh supports operations like our update method our schema method updates required schema is not our logging operation which supports the logging.
So we have four levels of logging detail based on whether and when you want those to show up. So we have like a fine level logging which only shows up while you're doing debugging and then we have info and the higher levels which will show up in your fivetron logs which we'll look at in a little bit. And then finally our operations which support our data operations like upsert, update, delete and checkpoint. So these are the four that we support.
Upsert is a insert or update based on primary keys. Update will only update and not insert. And then we have delete if you're working with a source that can tell you something's been deleted. We can mark something deleted in the target in data bricks if you as long as you just have the key for that record. And then finally, checkpoint, which is how we store the state of your connector in between. So, um, we'll go back to the restaurant example.
If you have an API that gives you orders, you might want to get all your orders going back to 2024, but then, so that's from 2024 to June 11th and then you need to save that June 11th. You save that with the checkpoint operation. I was going to mention on that too with that checkpoint maintaining the cursor. uh all those fivet connectors in the connector catalog zero CDC buildout at all. You do not have to build CDC and incremental sync.
That comes automatically with Fiverr. The only selection you make is like how often I want to go knock on the door of that source and get incremental changes. That's it. Give me one minute. Give me five every 10 minutes, 15, all that. So, just want to point that out. CL's going to show that on this as well with custom connector. Yep. Absolutely. So now we'll dig a little bit more into the operations and sort of the flow of the connector.
So our the required function that you have to have in your connector. py was the update function. So in the update function, we can do things like access the state. So we're going to get configuration, we're going to get state from Fiverr. So we'll talk about the state first. State, like I said, this is where you can store anything you want that you might want to use in your next next sync. It could be a time stamp. It could be a list of things that you haven't processed yet.
Um, it could be a different time stamp for every for a variety of sources. It's whatever you need it to be on the initial sync. It's always empty. So, we have coded in here what we want it to be in the case that it's empty. So, that's probably going to be like your initial sync start date. So, then the next thing we do is we access the configuration. So, configuration, what is that? So, while you're debugging it is JSON files.
So this is sort of the configurations that you can store in fiverran after you deploy. But while we're debugging we can play with these as different JSON files. So I have a debug file just has a few zip codes and then I have configuration. json which has more. But I want to de my debugs to run fast. So I'm going to have a little file. So we go back to our connector file and the rest of this is just sort of Python plumbing.
We're going to for every zip code in our list we're going to try some stuff. We're going to get our coordinates. We're going to get our forecast URL. So right here on line 49 is we have our first upsert operation. So this is where we are sending data to fiverr to write to our data bricks destination. So we only need two things. We need a table name and we need a dictionary. So we have our metadata dictionary which is just information about our zip codes.
So our table name, where does that come from? So we'll scroll down a little bit more and we get to our schema method. So all we have defined in our schema is table names, primary keys. What you're not seeing and what you don't need to do is define all 10 20 however many columns that are coming back from your API. So fiverr just like with all of our other data data connectors we're going to infer the data types for data bricks based on what you see in the dictionary.
So there's really no reason to define all the columns. You can if you want like there's a couple specific use cases where you might want to do that but it's really not required and it's one of the great things about connector SDK is that it's very flexible. So if your API starts sending new columns you start using a new feature those will automatically get created in data bricks. You don't have to change your connector SDK code in order to do that.
So we'll scroll back up to our update function where we're going through these zip codes and we're getting lat logs. We're upserting upserting zip code metadata go through the forecast and then we get back a list of forecast periods. So for we're getting like a 7-day forecast like uh twice a day forecast. So for every forecast period that comes back. We need to do a little extra stuff. We're going to add zip code to that forecast dictionary because the weather service doesn't know zip codes.
And then finally we'll do an upsert. So breakfast in there and dinner service. Exactly. There you go. Good way to think about it. So we're again just going to send a dictionary forecast dictionary to a table called forecast. We don't really necessarily even need to know what's in there. A little bit of error error handling here in case the weather service is not available. Um and then finally we do a checkpoint. So what we're basically doing here and it's maybe not particularly realistic for weather but this is how you save the time stamp you want to use the next time you sync.
So this goes into our cursor dictionary and then we we pass it to the state using checkpoint. So at this point I think we're ready to debug. So I'm going to go and I'm going to run my Fiverr debug. I'm going to pass in my configuration file. All right. So, we're seeing our logging. We have refined level logging, which we're definitely not going to need once we deploy. This is just too much. But it just kind of shows you what the connector is doing.
So, we're seeing the zip code metadata we get back from zipopotamus. And we're seeing that we have from weather. gov, we got the forecast URL for the zip code. We have 14 periods back. And we're we're logging all these forecast periods. And then at the end, we see that we did 75 upserts, two schema changes, one checkpoint. Great. Okay, so the cool thing about debugging locally is so far we haven't sent anything to Fiverr.
We haven't paid any cost for MAR or anything like that. So in our files directory under our project, we now have a new files directory. We have state. json, which is going to be your state that you've stored. So we're going to use this the next time we sync. We also have, and this is the cool part, is warehouse. db. So this is a duct DB file that you can open in a tool like DBver. Yes, which I do have. Okay. Yeah. Did you already have it set up?
I wasn't planning on it, but I can get up pretty quickly if you'd like to test it out. Yeah, let's take a look. Okay. Yeah, we got time. See if we can find it. And again, I know you had all this set up. I did have it all set up. So, sorry for the little extra here. Yeah. Let's see. Let's get out of that. Yeah. Just close. And you want to just do a new Yep. Well, you probably need me to find where this is, I would imagine.
Right. Yeah. Duct DB connection. So, duct DB. Duct DB. Yeah. And then you need the path to our warehouse. DB file. Yep. Let's go. You know, you have to borrow your colleagueu's computer and you're not quite sure where everything is. That's kind of what we're doing right now. All right. So, I will do that and then Perfect. Thank you. Got it. All right. So, all we needed to do is pass pass in that path to our duct DB file.
We click finish and then we have a connect a connection in DBver. This is probably impossible to read for you guys back there. So in our tester schema we have tables and we have our forecast table. So there we go. Okay. Sorry. So here we can preview our data. So we have our primary keys that we defined for the forecast table. Start time, zip code, those come first. We have all of our other fields that we didn't have to even really do any work to get these.
So Fiverr adds a couple columns to every table that we sync. Five trans synced time stamp that we wrote it and also five deleted which is going to be false until you change it. So we're happy with how this looks. Now we're going to go ahead and we're going to do our deploy. So nothing to data bricks yet. If you didn't like what you saw in that warehouse, you could come back, make your changes in the connector. py, test it again.
No no cost being incurred at all. So that makes it really nice. I think it's a very handy feature. Yep. So now we are going to run our deploy. So this is a simple uh command and we'll we have all this in our documentation of course but uh has built this awesome shell script. So fiverr account name uh destination name Azure data bricks. Yeah and there's you have it this is connected up to an API key within our fiverr account that is associated with that MDS data bricks hands-on lab account name.
So we have the API key sitting in the config JSON or environmental uh file. Um but those are that's the three things you need. I need my account with the API key. I need a destination. Most of our customers have multiple destinations in their Fiverr account. And then co you get to name the connector whatever you want to and in this case we've chosen to name it DAS demo 611. So uh when it deploys it's telling us what it's doing.
So, it's going to send the conne files that are in your project directory. Any Python files will get sent. Uh, it's going to send a requirements. txt if it's been created, but our connector doesn't. If you're curious about Python libraries, we have most of the basic ones pre-installed. And when we do when we deploy, we use piprex to compare your imports against our standard ones. And if there's any sort of delta, we make a requirements file that gets sent.
And then we send the content of configuration. json. JSON. So that's important because I kind of glossed over this. This is where you would also send in any API keys, stuff that may be confidential. I also use it a lot for things that I might want to change without having to redeploy my code. In this case, my zip codes. So this has been deployed and we get a nice hyperlink down at the bottom. And yeah, you should be if you just alt or not alt but command should escape out of here dashboard.
All right, great. Okay. So, we're gonna go and find our connector that we just made. Okay. So, it's ready to sync. It doesn't kick off a sync automatically. You still have time to back out if you decide you don't want to do this. So, I'm going to kick off this initial sync. And so, this is going to take a couple minutes to run even though it's not very much data. And I'll tell you why in a minute. So, let's go back. I'm going to look at a connector that is already done.
Okay. Okay, so this is a similar connector. So let's just take a walk through some of the tabs. But first, if we hover over, if you're looks like about half of you aren't familiar with Fiverr. Uh we zoom in a little bit. So this sync started at 90934, ended at 911. So one and a half minutes. What is it doing in that time? It's not very much data. So extract time 10 seconds, process time 4 seconds, load time 12 seconds.
So that's 26 seconds. The other minute and change is us spinning up a container in our Google cloud platform. So we create a container, we install any packages that you need, then we run your code. The rest of the data processing time is what's mostly determined by your code. So this is sort of magnified and we have a small amount of data. But if you're loading a large amount of data that runs for 10 minutes, it's going to be a minute to spin up the container and then 10 minutes of processing your code.
And as a customer, a fiverr customer, I don't have to determine container size, anything like that. Everything you're describing is under the under the covers. Yep. Exactly. So, uh, with connector SDK, there's an extra tab up here. If you are a Fiverr user, you might recognize that this is not a normal tab. So, you can see any of the logging statements that you've added to your code. So, we have all of our info level and warning level logs here.
And these are here in your UI for seven days. If you use the Fiverr platform connector, you can write it to your your logs going into your data warehouse. keep them for as long as you want. We have our schema tab which acts like other fivet connectors here. If you decide you don't want some of these columns, you can uncheck it. It'll walk you through a dialogue for how do you want to handle stuff that's already been written and then usage and finally setup which I mentioned a little earlier.
Anything that's in your configuration file gets uploaded to Fiverr and you can change it later without redeploying. This is where you do it. So if I click edit connection, I can go in. I can't see this. This is why API keys good place to put them. You can't see them, but you can overwrite them. So if you have an API key, someone leaves the company, you need to cycle it out, you can replace that in the UI, you don't need to redeploy your connector.
So going back to that diagram, CL, we had that one box that you know, as a as a customer, I'm responsible to code up and then you have those other two/irds. You're you're really in the twothirds, right? Exactly. Yep. Like I'm not you're not having to code up all this orchestration. And I think on that setup, I mean that looked to me like classic fivet. Like I can say I want a minute incremental sync polling time. I want five 15 whatever that is.
This is no fiverr. This is this looks identical to really any connector out there. So I can go a minute uh up to 24 hours on my sync frequency. Yep. Exactly. So uh I mentioned earlier our documentation. We have a setup guide which is sort of a step by step how to set up your first connector, virtual environment, pip install, etc. Um, in here are linked our GitHub repo. So, we have quick start examples which are and we're getting close to the end.
Um, we have quick start examples which are sort of this is where your hello world is basic uh five connector SDK 101. We have our weather byzip code example which is what we just walked through. So, the code is in here for these quick start examples. We have common patterns. How do I authenticate based on different patterns that the API uses? API key, HTTP basic, OOTH. So, and then finally, we have the source examples work uh directory, which is fully functional connector SDK connections for sources that Fiverr doesn't currently support.
So, for these, you're probably going to need an account for whatever source you're trying to replicate. So, at this point, I see I was just going to ask you on this because I I'm a big fan of this example repo, but my friend, who I talked about earlier, who wanted the Java version, he he said because I suggested I said, jump out there to Claude. Claude will help you on this. There's a lot of ways to do this. Uh, I've got uh model assistance.
I've got this example repo. Code it from scratch. What What has worked best for you as you've been building the 40 to 50? What's what's been working really great lately is I'm a fan of Claude. Okay. I like I like Claude. Um chat GPT also great. Cursor uh IDE does a pretty good job as well. So we actually have a directory AI and connector SDK where we have like a Claude instructions file so you can feed Claude. These are the important things to think about.
Um they're great for like a first draft and then sometimes I have to iterate if there's some nuance that it didn't quite get. But the use of AI has greatly speed up the time to create these connectors. Um, so we actually have a blog post uh about how to create a connector in under an hour using claude. So that is out there as well. Yeah, we'll show you how to how to basically clone our repo all those examples ingested into claw to become really that foundational knowledge base and it just becomes super easy to build this type of connection.
Yes, there's a lot of different ways to do it though. All right. So, at this point, our DAS demo June 11th is done and it ran in 2 minutes 18 seconds. Uh, extract time 30 seconds, process 11, load time 15. Um, it processed about 20 zip code, so more than our more than our debug. So, I'm going to hand it. He's going to show you how everything works in data. Are first of all, are there any questions on what CL took you through here? because this is we uh everybody who sees us go oh wow there's a better way of doing custom connectors now any any questions on this piece yes about the schema sure yeah so um the schema change for connector SDK works in much the same way as schema change for any other five connector so um our data type inference we try to pick the most specific type that will loss list loss lessly accept the data type that you're trying to write.
So the first time you send a column from the API and it's an integer, we make the type an integer, but then the next time maybe it becomes a float, we'll change it to a float in data bricks. Um, depending on the destination they allow and don't allow different types of changes, but we try to what we usually end up doing under the hood is I believe we create a copy of the table, do a cast operator, put all the new data into that table, do a rename.
So that's sort of how we take care of it in general. Uh there's no you shouldn't it should happen without you knowing that it happened for the most part. Yeah. Question here. Um so for this demo is there a way to see like the total cost? So for usage, there's usage isn't here yet for this example because I think it updates like hourly or something, but and so for all your initial syncs with Fiverr across all of our connectors, including anything custom you build, the initial historical sync, whatever size that thing is, is free.
No charge at all. And then you get 14 days of incrementals for free. during I think right around day seven or so you'll start seeing in that usage oh here's what my incrementals are to start getting an idea of what your consumption pattern would be for that connector. So yeah, it's a little bit early on this one since you just created it, but really really good question. You would get that after after a few days. You'd be able to see kind of what it would be for.
And is that in dollars and cents or is that in MAR? Well, I wasn't going to bring up all the details around it, but basically we charge for updates to a row. If a row updates one time during the course of a month, that's a consumption event for us. It could change another hundred times or thousand times. Doesn't matter. Uh we call it monthly active rows mark. Great. But yeah, you'll see all of the details uh in there.
So yes, total more free. Yeah, this is the other connector. So yeah, so and since we're yeah since the it all depends on how the primary keys are defined basically. Yeah. What gets considered a mar? Other questions on connector SDK? API key. Uh just the connector name. You just name the connector. Yeah. And on the on a regular deploy, I put a little bit of scripting around that one just to make it easy so we can default on a regular deploy command.
CL if uh Yeah. Scroll down to debug. Yeah. So deploy. Yeah. Yeah. Here we go. Yep. So this is a standard deploy command configuration. So a little bit about configuration. json. It's required the first time. After that you can deploy without the configuration file if you don't need to make any changes. But the first time you deploy you are going to want to send the configuration. So other than that you need your Fiverr API key which comes out of your Fiveran dashboard, your destination name, an already existing destination.
We're not going to create it as part of this process. So it has to already exist in Fiverr. And then the name of your brand new fiverr connection which is going to be the name of the schema in data bricks. Other questions? I know we got I think three minutes ramen left. Is that right? Yeah. Yes. Go ahead. Might be a little off base but you mentioned earlier that changes. So that's a great question. So um most of our use cases initially were APIs where they're giving you like there's a update time stamp you can pass to the API for relational databases as of now we're still based on sort of relying on an audit column in the table and doing select statements.
Yeah. And we have a tremendous number of database connectors out of the box as well. There could be use cases though where you'd want to use a connector SDK. Good question. Anybody else? I know we got a couple of minutes. I I did want to and I I'll try to do this super quick. I know we got two minutes and then they are getting us out of here. Um let me go out just real quick because I did at least want to show you the data here as it landed in data bricks.
What was it called? CLI DIS demo. Okay. Everything lands in a in a Unity catalog. Was it DAS demo DAS I will just scroll down instead of demo. Yep, that's it. That's it. There's your two tables. Forecast zip code. And if I take a little sample data view there in uh data bricks data intelligent platform. I got that. I was going to read. So I'm a big fan of data bricks apps. Has anybody built a data bricks app yet? Oh, they're awesome.
You you got a So you got you got five different flavors in uh data bricks to use. Let me actually I'm just going to we'll do this just real quick. Last thing here. It's it's a little bit hidden. It's under compute. If you go up here to apps, I'm not going to create a brand new one. You got five different options. I got Streamlet, Shiny, Radio. Uh all this is there for you. We built a cool little weather app for our restaurant tours in the audience called Weathermind.
And uh you can build this in cursor, VS Code, deploy it with Datab Bricks. Data Bricks gives you a really nice little sync command up there and then a deploy command. Uh we've already got this one deployed. I'm not going to go ahead and change to the data set that you had because we just don't have time. But this will give you an idea. This is a streamline application hosted in datab bricks. I've got security governance of data bricks around this thing.
And uh see what is Oh, I happen to live in Cypress. I know for a fact it is raining cats and dogs, pots and pans, whatever you call it in your world. uh cypress delivery today. I'm not going out for that. Best patio location is my hometown of Gaylord, Michigan. There you go. You could got sunshine from hometown of Claude Shannon. So that's kind of fun. And then uh if you're not using data bricks apps to make uh data bricks AI calls as well, uh we've got like I think three or four little micro Genai apps that are going out in this case to claw 37.
Data bicks has dropped 40 in into data bricks as well. That's all available. Uh, last thing I want to point you to, we really appreciate everybody. We have, if you want to get hands-on with this yourself, this QR code right there, uh, click that and you can join us for a virtual hands-on lab to try out the connector SDK. See, I don't know if you'll be hosting that or I'll be on it, but there'll be somebody from Fiverr that will uh, take you through if we're not there.
It'll be fire. It'll be fun. Yeah. I'll leave that up there. Thank you everyone. We really appreciate you. Thanks everyone. Thank you so much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.