Skip to main content

Can you even build Databricks entirely out of Open Source Software?

Summary

  • Open source stack available: Databricks' core components are available as open source—Spark, Delta, Iceberg, Unity Catalog—enabling teams to build functional data platforms using only open source software.
  • Lakehouse at Home project: This video walks through building a complete lakehouse environment using open source tools including Kubernetes, Trino, DuckDB and other projects maintained or supported by Databricks.
  • Cost-effective alternative: Discover which open source projects provide equivalent functionality to commercial Databricks offerings, enabling flexible deployment options for data and AI workloads.

Can you even build Databricks entirely out of Open Source Software?

Watch: Can you even build Databricks entirely out of Open Source Software?
For years now, Databricks has been charging for open source software, so we decided to build our own with the help of Lisa Cao, Open Source advocate at Databricks.
Check out Lisa’s Lakehouse at Home project here: https://github.com/lisancao/lakehouse-at-home
Links to all the open source projects we discussed:
MinIO: https://github.com/minio/minio
SeaweedFS: https://github.com/seaweedfs
Delta: https://github.com/delta-io
Iceberg: https://github.com/apache/iceberg
UC: https://github.com/unitycatalog
Polaris: https://github.com/apache/polaris
Lakekeeper: https://github.com/lakekeeper
Gavatino: https://github.com/apache/gravitino
Spark: https://github.com/apache/spark
Trino: https://github.com/trinodb
Flink: https://github.com/apache/flink
DuckDB: https://github.com/duckdb
Airflow: https://github.com/apache/airflow
Docker: https://github.com/docker-library
Kubernetes: https://github.com/kubernetes
Jupyter Notebooks: https://github.com/jupyter
MLFlow: https://github.com/mlflow
#OpenSource
#DataEngineering
#Lakehouse
#Databricks
#OverArchitected

Chapters

FAQs

Can I really build a Databricks alternative using only open source?

This episode shows how most Databricks functionality has open source equivalents. While architectural complexity exists, the Lakehouse at Home project demonstrates a working implementation.

What open source projects are included?

The stack includes Spark for compute, Delta and Iceberg for table formats, Unity Catalog for governance, MinIO for object storage, and Kubernetes for orchestration.

What does modern open-source data engineering look like?

This video explores building a complete data platform using open-source components alongside the Databricks Data and AI platform. Modern open-source data engineering combines tools like Apache Spark, Delta Lake, and Apache Airflow for orchestration to create a fully-functional data stack. The video demonstrates that many foundational Databricks capabilities are built on or compatible with open-source technologies, enabling organizations to leverage best-of-breed open-source tools within a managed, governed platform.

Why would you use open-source software for data engineering?

Open-source data engineering tools provide transparency, community support, and cost flexibility for organizations building modern data infrastructure. This video shows how leveraging open-source components like Apache Spark and Delta Lake on the Databricks Data and AI platform combines the benefits of open ecosystems with enterprise governance and managed infrastructure. Open-source foundations enable organizations to avoid vendor lock-in while accessing cutting-edge technologies that are continuously improved by broad communities.

What does the modern open-source data stack look like?

The modern open-source data stack, demonstrated in this video, combines Apache Spark for distributed processing, Delta Lake for ACID transactions and governance, and tools like Apache Airflow for workflow orchestration. When deployed on the Databricks Data and AI platform, these open-source components integrate seamlessly with managed services, Unity Catalog for governance, and AI capabilities. This architecture enables organizations to build scalable, cost-effective data platforms without sacrificing governance or reliability.

Full transcript

0:00 Intro

Hello, hello, and welcome to Overengineered with and, where this month we quit. That's right. We had enough of Databricks, and we've decided why should we pay all this money for Databricks when in fact we could do an open source version completely for free. So, we've sent those emails, we've burned the those bridges, we're not being invited back, and so now we're in the scenario where we need a platform to be able to do devrel stuff about.
And to do that, we're going to try and build one out of open source with some wonderful help from, open source uh aficionado, enthusiast, absolute juggernaut, wonderful contributor in the community for so many open source projects. Welcome. Hi, thank you for having me. I am really excited. It's nice to be here for maybe the last episode of Overengineered, I don't know. I think they're coming to get my laptop right now, so we need to finish this fast.
I actually did go down this rabbit hole a little bit ago. So, for those of you who don't know, I'm a devrel here at Databricks, but I focus primarily on open source, and so I really like playing around with open source technologies. Cool. Okay, then. So, yes, tell me. I'm I don't know where I get started. Help me, please. I have no job. >> Okay. I will do my best, valiantly so. Um so, the good thing is that Databricks pretty much at most points of the stack does have an open source equivalent, even like maintained and built by Databricks themselves, which means that it's actually pretty feasible to rebuild a lot of Databricks, at least in terms of its core skeleton, without necessarily having to go to anything proprietary.

0:54 Setup

The rest we can actually supplement with other open source technologies that even Databricks uses as well. I'm going to caveat this by saying nothing here is uh proprietary. I am running Ubuntu, I am using Vim, and we're going to use pure Vim for this uh because you guys are now going to be open source people, and open source people love uh their minimal text editors. And we're going to be working through it. So, I have a open source repository called Lakehouse at Home that you can use if you want to follow along with this.
Uh and you can uh get clone it for free, uh which is awesome. But, one of the things that you'll need to do if you have a blank machine, so that's probably the step one is that you need a a computer, which it doesn't sound like will have for very much longer, but we're going to we're going to work with it. And the second is that you need to have an operating system and some files and some data. I'm not going to tell you which data to use, that's really your choice.
But, what you will need is an object store. There's a couple of different open source options that you can use here. The first is, you know, either MinIO or SeaweedFS. I'm using SeaweedFS, which is S3 compatible. I'm already scared. Yeah. I like how you started with the real basics of, you need a computer, which I got to say >> repository looks like currently. So, we have a bunch of our DAGs, we have our configuration files, our data, demos.
Uh Docker is going to probably be your best friend here, uh especially if you're running a lot of open source technologies, it helps manage a lot of different dependencies that you'll have. And also, you'll need to actually download uh the source code for a lot of this, which will often times come in a JAR file, especially if you're working with a lot of Java. And then, we're going to look at some of also these different uh services.
Is that too mismatched,? Do you want me to slow this down so that we just walk through it? No, no, no, this is fine. Uh all I'm thinking is, oh my goodness, I've not used Docker in years. Um and now I'm going to have to use it again. >> >> But, yes. So, yes, I'm deploying the entire platform. I actually have all of these different services running, so I built a CLI tool around this uh called Lakehouse, and this is just the output of Lakehouse status.
And you can see that all of these different services run on different ports on my machine, and you can test how they're connecting to each other. So, networking is a really big part of data engineering, actually, because you have so many different services that need to connect to each other. So, we see that the Airflow web server here is not working or live. So, if we look at the rest of this, we have our Terraform recipes, we have our Docker Compose files, we have our JAR files, we have different schemas and scripts and data.
And if we look at the different Docker Compose files themselves, we can see exactly what goes in here. Uh in this case for Spark 4. 1, we have it pulled from the Spark image. So, if I take a look at the rest of these, though, you don't need to know all of these, I already have them figured out, and most of the open source projects we work with will already provide you a Docker container. I'm sure you're feeling the open source now.
I'm feeling it. I'd like to like Does open source come with like a nice UI? Like so that I can just I'm feeling it. I'm not seeing it. So, again, yes, but UI is not really like an open source-y type strength, uh which is why a lot of the services you use have nice UIs around them, including Databricks. But, as I was showing there, we need to connect to a lot of these different services together uh really quickly, including like different credential files and configurations, which we'll get into later.
So, wait, I have some questions before we move on. So, sorry, that's that's just like a config that runs in the background. There's not a lot that like I need to set up, cuz I There was a lot of words that you said there, which I kind in passing. Do I know run into this? >> often times there's a lot of work in between that requires glue, right? Sorry, as in code glue, not AWS glue, right? >> this in a little bit, too, but often times, say if you're running like the newest version of Apache Iceberg, like 1.
10, that has different dependencies that maybe don't work with the different Spark versions that you're running, uh which maybe don't work with the different S3 versions that you're running, or Hadoop versions that you're running, that maybe don't work with the different Kafka versions that you're running. And so, often times you'll need to have dedicated um JARs that are just meant to connect to services together or pick and choose between your dependencies.
Yeah, I understood that one. So, I have this script here that shows us how Seaweed works. So, if you're running Seaweed, anything that's S3 compatible will work with Seaweed. It just provides you an object store where you can have like your same buckets and things like that. Uh same S3 compatibility, just on a local system instead. Nothing too crazy running on my system. You can scale this out, of course, depending on what kind of resources you have.
And we can also take a look at some of the Parquet files that I have stored here. So, this is in my bucket Lakehouse, and we have all these Parquet files, uh which is pretty straightforward. Uh ignore that error, I do, and it seems to work fine. Uh that is the key of open source. >> what actually is required to set up Iceberg. Essentially, if you have a bunch of Parquet files, those are just the columnar files, right?
You still need the metadata around it to actually make it a table, and that's where Iceberg comes in. And there's a lot of different other table formats you can run to, too, like Delta Lake, for instance. Delta Lake is really good, as well. Uh it really depends on what your flavor is. Both are very similar, and we can definitely get into that uh in a little bit more. So, are there any questions that I can answer while we're running through this? , I think at some point you and I are going to have to start diagramming to say how we're going to stick all of this together, because I am not a CLI person.
What do we Yeah, catch us up. What are we What are we show What are you showing us now? So, you have your object store now. You have your buckets. And then you now have a bunch of Parquet files that you're throwing in there, right? So, that's your raw data. Typically, an engine can just read directly the Parquet files, but it's a little bit messy, because, well, you don't really have any metadata. You might have different locking issues, things like that.
And so, Iceberg provides a metadata layer on top of this. What it'll provide are the metadata files and layers that engines can natively understand, and that really helps different engines coordinate across their different rights. If you have multiple engines, it makes it much easier to coordinate, and actually helps preserve the integrity of your Parquet files by providing you things like snapshots, metadata statistics, you know, metadata files.
There's a lot of different complexities at scale, especially if you have not just hundreds or thousands, but millions and billions of Parquet files with different engines writing to them. Having all these different rights, and you're worried about the integrity of the actual Parquet files. If you're using Spark with Parquet alone, you're directly loading all of those Parquet files straight into Spark, which is not clean hygiene.
And so, if you have Iceberg in between there as a table, then you can preserve the integrity of those files and make sure that there's no like abandoned rights, orphan processes, things like that, and that actually makes it much cleaner. There's a lot of different options if you're coming from, say, the Delta world for this. So, the first is that you can use something called the Delta Kernel to ensure that your engine, so to explain, the Delta Kernel is an open source project that is a Rust native and embedded library that you can actually embed within your different engines to be able to understand the Delta protocol natively, which is very similar to how compatibility works with different engines when it comes to the Iceberg rest protocol.
The other option is to use something like uniform where you can actually use Iceberg clients to read Delta tables as Icebergs. So, does that make sense in terms of how the table formats and the engines and all these things work together? Yeah, I mean I mean we've done a lot of Iceberg and Delta stuff as part of Databricks anyway, so I'm kind of familiar with like a lot a lot of the the kind of um what the features are kind of on the box and like on the marketing.
Um I guess from a open source perspective maybe something I'd not considered is if we've got multiple engines doing different things. So, if we've got Spark doing something and then we've got Flink doing something and then we've got something else doing something. Ensuring that they all work together is probably something that has been obfuscated away in Databricks, but in open source land you have to think a lot more about.
All right, let's take Can we Let's pause. I'm taking a note from. We should start drawing something because my head can't fit all this in. >> Yeah, we have a seaweed. Seaweed's getting a huge push from us. This is just like a random choice. It's actually >> this is the first thing that we said, you know, so my uh storage but is this going in here? >> perfect. Throw it in there. We're I love it. This is good. Good job.

10:51 Base diagram

Throw another one. Throw it Let's copy this, copy that, MinIO. Minio? IO? Is that right? Table formats. Delta. Iceberg. Okay, so we're building it up. I like where we're at. We have engines, right? And so wait, explain this Delta kernel thing again just very briefly because that's where I got I mean I got a little bit lost. Right, so we have like something like Spark. And doesn't Spark use Didn't they migrate it to like also use Delta kernel underneath like the Spark connector for Delta uses the Delta kernel or no?
I think where Delta kernel really shines is in engines that have nothing to do with the Databricks open source ecosystem. So, let's say if you take something like Flink for instance or Trino, uh DuckDB, etc. that don't necessarily natively understand the way that Delta works, especially when it comes to its metadata systems and things like that. They don't need to build that in anymore directly into the engine. They can just instead take Delta kernel, which is like a very low-level embedded library, stick it in, make it compatible, and then never have to worry about it again.
Oh my goodness, what have you done? The Delta Delta kernel helps the engine, but it doesn't it's not part of the engine. Like It's like an inline module, I guess. I don't know how to It's like a plug-in. It's like a plug-in. You also said Trino as well, so let's add that in. I'd put the arrows the other way if I were you, but it's fine. Also, you called the table format Delta RS, but Delta RS is like the connector, not the not the format.
I thought that was the implementation in Rust rather than Anyway, fine. Anyway, it's fine. >> We can argue about this later. Does this Does this accurately represent what you've described so far or not? No, not at all. It's actually completely I'm I'm joking. I'm joking. But there are some tweaks I would make to it. So, like said, Delta kernel really lives on engine side compared to being in the table format. So, Iceberg has the IRC built in, whereas Delta can have the Delta kernel embedded in an engine for instance.
Okay, I'm just going to put them in between to show that it's kind of the glue to stick them all together. How How about that? I might get a little bit picky about that arrow direction, but I think you're good. Ah, This is a beautiful diagram. I also want to note for you guys in particular uh if you're building a data platform for the first time, I would not start out with two different object stores, two different table formats, and three uh engines.
Fine. You know what? Well, this is our platform and we're going to do it our way as well. Hang on, so we have storage, but we also have like storage uh slash own computer. >> >> So, if you really want to get pedantic about it, I've tested this on my machine on like So, my machine's pretty powerful, but I think you can run what I have of like 16 GB of RAM. Setting the stage realistically, do you The only reason you'd want to run it on your machine is simply to emulate what it would be like at scale, right?
Because like there's really no reason to set any of this up for a single machine. Or is there? Uh it depends how crazy you are. If you're like me, like kind of a little bit, but I think you can actually build this in a business. Like there's a lot of businesses out there that are very invested in open source. And so, they would take a framework like this and then also apply it within their own servers on their own machines with a lot of other overhead, too.
But if you're serving it to like say actual customers, you probably want to keep it off your laptop. Okay, fine. Let me update this to say not on laptop. Uh >> confused. , diagram. But now you know where we stand, so you pick it up from here. So, I'm actually going to make you shift everything up between table formats and engines. You spent a ton of time building this beautiful diagram, but I want to make space here for a catalog and explain why we actually do need a catalog.
So, I went over this a little bit in the demo and we we can go back into it soon. But generally speaking, like I said before, you didn't want your engines directly reading Parquet, so now you have them reading the table formats. And now you have every single engine here configured to read the table format separately. We're going to simplify this by adding an actual dedicated catalog implementation. In this case, we're going to use UC OSS, which is the Unity Catalog open source version.

16:09 Catalogs

A catalog in this case will provide a unified interface for engines, but it will provide us something that's really important, especially if you don't want to get sued in your new business endeavor, which is data governance. So, a catalog will help broker and manage and tag all of your different tables here and then actually control what queries can access which data as well. I may have been overly ambitious and just wrote Unity Catalog.
Are there other catalogs that we should consider? Or is Unity Unity the one and only? Catalogs vary quite widely in terms of implementation. The Iceberg rest catalog is pretty basic in terms of how it couples to the data format itself, but other, you know, different options that you have there in terms of catalog implementations include things like Lakekeeper, which is a Rust native Iceberg catalog implementation. There's also Polaris, which is an Apache project as well.
There is also Gravitino, which is another open source project. And catalogs over time have really changed its definition. So, we're talking about specifically catalogs for open table formats, but other governance catalogs like Informatica or Collibra, you know, DataHub, you know, those are all examples of catalogs that kind of predate or at least extend around the Iceberg ecosystem. One of the things that a catalog does is that it actually can coordinate different commits.
So, this is something that was introduced in the recent version of Unity Catalog OSS. So, the reason why this is so important is because you might have Spark and Flink both writing to the same table underneath, and this can kind of happen in parallel, but all of those changes or all of the metadata files that might be, you know, generated as a result of that might differ from each other creating a really messy situation, and you might just not know what your source of truth is.
You might also want to just in general account for more concurrency. So, something that Unity Catalog can do now is actually commit coordination. So, if you have multiple engines committing and do querying the same data set, what Unity Catalog will do is take those commits, serialize them, and then act as the writer in that case directly to the actual table format. This is also really cool because Unity Catalog also does the credential vending.
If you're not using a catalog, these credentials are often hard-coded into the engine itself, and it's just sort of like, "Okay, whatever this object store is, you can access all of it as this engine," which is not super scalable over time, and you might over-permit something to access things that you don't want to. So, instead what Unity Catalog does in this case, and many other catalogs do this as well, is that it will actually provide temporary credentials to the engine for that single query so that then you don't have to worry about it at all.
Does that make sense? It's overwhelming. Yeah, there's a lot of decisions we got to make. I'm used to just clicking on a thing and then a box comes up saying credentials and then you just do some stuff and then hope for the best. It is really complicated, especially when you have like a ton of engines, a ton of users, but again, unless you have really strong vertical integration with your governance, a single catalog's not going to be able to do it all without needing like extra workarounds or just like extensibility around it.
Uh and when you keep saying vertical integration, do you mean like vertical across all our boxes here there or like or does vertical integration mean something different? It does mean across all these boxes, but there's a lot of boxes that aren't included in this. So, if I want to look only at the lineage between like Delta tables or Iceberg tables here and see like how a table has transformed over time. That is pretty straightforward, but I might have like a Tableau service or an Excel sheet somewhere that also has data transformations that occurs that is not tracked in any sort of way.
And that lineage is completely lost unless I do this like manual work of like actually building the code in between that. So, if we want to set up something like Unity Catalog, one, we're going to have to run it as a service just like everything else here is run it at run as a service. And once we do have that, we can actually configure it to detect and catch our tables. So, in this case, we have it set to detect our Iceberg tables and we can actually then use and query let's say like Unity Catalog either using like a smart query or using post or you know, rest or whatever to be able to actually list and interact with our tables instead of going directly to the Iceberg rest catalog or worse going straight into SeaweedFS and trying to list individual parquet files.
You can see that the return is in a JSON format and we actually get our catalog. So, if you're used to working with UIs, this is the same information just not in a drop-down menu. And earlier you said something about setting this up as a service. Like, what does that mean? Sorry, you said that like that's one of many options. Typically a service is something that's running 24/7 that you can then query. Like, if I want to go to a website, it's always going to be up and it's always going to be there and it lives at a very specific place on my network.
There is a lot of networking that comes to data platforms that people do not realize. >> my existing catalog so that then I can see it listed out. And you can see it's pretty clean. Cool. So, we're having a look at this data. And is this meant to be uh like a substitute for UI of like one of the things that I can browse and look through or is this something that other services are then going to query and come back with that information?
They're going to query and come back with that information. Uh I think only a masochist really interacts with their data infrastructure in this way uh to be honest. Unity Catalog does offer a UI. Most of these services do, but I will also say that most of the UIs in open source are typically stripped down compared to their proprietary uh yeah, proprietary equivalents. What is the like go-to like um for notebook interfaces?
Is it still like Jupiter? Like, what's the what's the latest and greatest? I think Jupiter is still pretty much the winner in this space. It has the most robust ecosystem. So, if you want to work with your data transformations in that way, that's really great. Jupiter is hard to run though. Like, there's a lot of dependencies that make it really messy to run at scale, especially if you're managing the notebooks itself, but it is possible.
You hear that,? It's possible. >> actually how to set things up in Spark. So, this is probably one of the things that is really complicated for anybody running open source Spark and has never had to deal with it before. So, Spark has a lot of configuration files that makes it really hard uh to actually understand what is going on. And the way that Spark understands whether or not you're, you know, using an Iceberg rest catalog or Unity, how it has all these different credentials is through these configuration files.

23:06 Setting up Spark

So, you can see that there's Hadoop configurations with S3. We have to load in our actual JAR files in here that has all of that glue that I talked about before. So, things like Spark runtime, you know, AWS compatibility which we're using to interface with Seaweed, and then all of these extra bundles as well. So, it's really important to note that this is probably where you're going to have a ton of stuff set in there.
I have it with my Docker container so that my Spark versions know which configurations to pull from, but it's really important if you're setting up for the first time that this is something that you keep an eye on. Cool. And if I get it wrong, it just doesn't work, right? Yeah, everything breaks and your internet gets canceled. So, one of the other things that I want to show you guys is how I'm using Spark 4. 1. So, if you recall earlier I had Spark 4 and Spark 4.
1, I'm using 4. 1 for most things because it comes with a lot of awesome new features. And typically the way that you submit your transformations or your code uh is through something called Spark submit. So, I guess in this environment like everything that you're setting up with infrastructure is also going to be in the same kind of terminal window CLI tool that you're using to do all your kind of uh data analysis in your your ETL.
Yes. Precisely. And so, all of this runs on my home system right now, but you know, the Spark is up, everything else is up just in terms of services on my machine. And it's exactly the same as what you would use in like a cloud system. It's just where it lives instead. Sure, but if I had a bunch of users and I didn't want them to see all of this stuff in the background, I'd have to figure out a way to make that happen.
So, this is a really great segue into our next part, which is if you're actually planning to build out a platform or if you're trying to scale out your actual cluster. This is all single node, which is pretty overbearing. But again, like said, you're going to probably have many different users. In that case, I would really recommend using Spark Connect. So, Spark Connect allows you to have lightweight clients that users can interact with Spark on on their local machines or wherever they are.

25:19 Clusters Scaling

And then you manage a larger Spark cluster and then they just submit jobs to that and there's a very easily ensured compatibility between the two in terms of protocols. It's very simple to set up. You can use either PySpark uh to download it or if you want to, you can even just download the client itself. And it's really straightforward and we can get into what some of that code looks like really quickly. And you can see me moving through these directories that don't exist.
Uh Often times on Databricks you will not have this issue. >> really over optimi- like over estimating my skills uh or you're delusional on how well I can do all this stuff? I think it's much easier once you get the hang of it, especially with AI these days. You can actually do a lot of this glue work and debug more easily. But this used to be a really tough thing to do uh especially back in the day. The fact that we have it running on a laptop now is pretty like insane.
With Spark Connect, what you're going to be able to do is again submit jobs to like a remote instance of Spark. And it's really nice because the dependencies, like all that configuration and all of those JARs and things like that I showed before, your client actually doesn't have to worry about any of that uh if you're using Spark Connect, which is much easier. But if we look into how actually Spark Connect works, we have our localhost here, which is where uh the remote is set and you're going to see it denoted by the SC namespace.
And then we're actually going to be able to show you some different ways to submit Spark jobs. So, again, because I have a Docker container running Spark, a lot of these commands seem more scary than they are. But we're going to be able to create our Spark session here and then I'm going to show you really quickly how to actually submit a job and get some data out of it. We see here that we have a Spark Connect uh demo that's written out and we're we have our Spark SQL queries and we're actually going to submit them uh through Connect and we can see the output of these tables.
So, it's pretty straightforward to run. I would not again recommend running it locally on your own machine, but you know, to each their own. Yeah, so I think we've got a few things missing from our architecture diagram in terms of where to actually run this stuff. Like, we've got some storage here, but what we don't have is any clusters or anything set up like that. Where does this cluster live? Sorry, you So, a cluster is just a collection of machines.

27:58 Clusters Diagram

So, technically it's somewhere on the network. I think that's fine. Okay. I guess it only applies to Spark, not Flink and Trino, but that's okay. There's no way we can like make it look pretty, I think, at this point. So, that's okay. >> on. And so, sorry, were you suggesting to have like Spark Connect like this and then have uh a Jupiter cluster? >> I'm so confused. That does look like a cluster of something. >> reason why I say this is because the way that you install like Spark Connect, at least if you're on Python, is by installing like the Python package.
And in the notebook as well as where you like identify where you're going to submit it in your Spark session. Code. And then so, maybe then cluster can be above even higher. I think we can just remove cluster, honestly. Let's just pretend we have like, you know, compute ephemeral like just available via Kubernetes somewhere or whatever. Sure. Okay, yeah. That happens magically. I like that it just says CLI. >> >> I just need something that's going to remind me that this is what I'm going to spend my time looking at and I'm just kiss goodbye to a lovely little UI that I'm used to. >> the least and it's only going to get more complicated in a little bit.
If you're running Spark Connect compared to normal Spark, so normal Spark will have your code in your notebook and then it'll also have, you know, your actual Spark implementation such engine which is like your JVM and then all the drivers and that all is like one process that's coupled together, right? And it's all very it very hard to separate it out. So if you're using Connect, what you can then do is submit over a network protocol in this case GRPC to connect to a server.
And same like sort of concept that we talked about with Unity earlier and so we're connecting it and we're submitting it to a service that's running somewhere else and then we don't have to think about configuring all of that locally. Does that make more sense? I'm sure with time it will, yeah. >> everything, it's just configured a little bit differently especially if you're running really large organizations. We can move on to the next part of this now which is if you're developing a lot of Spark code, yes Connect makes it easier in terms of like somebody else, in this case you guys sadly still, manage all of the actual Spark glue code and the configurations, all of these different things that jars and dependencies.

30:54 SDP

But there's also a lot of issues that often come up with Spark code itself in terms of the way that you actually set up your Spark session, the way that you handle parallelization, the way that you handle retries. There's a lot of sort of code that goes around the transformation itself which if you guys end up having customers God forbid, >> guess we used to STP from a Databricks perspective. Uh so if it looks very similar to that, I think we kind of understand that bit.
It's more the setup that we're struggling with. Okay, cool. That's awesome to hear. So Spark declarative pipelines comes with a different set of tools than maybe people are used to when it comes to like Spark submit or running their own jobs. I won't get into it too deeply, but when you initialize an STP project, you create a YAML file spec that then contains your different configurations for your different files that includes your code and then you can then perform something like a Spark pipeline dry run and then submit that into Spark itself.
So you can also do a normal run. It really depends on if you want actual data to be processed or if you just want it to like say double check your logic for you and make sure that you don't really have any like issues. So assuming that you guys are unemployed right now and you want to save money on compute even if it's your local compute, I would highly recommend using dry run very liberally in your case. So you can see that STP syntax requires declaring a table and we have the ability to use streaming as well as our normal declarative batch syntax as well.
So we have this table here that's, you know, actually pulling from our Kafka service that we set up earlier and is pulling out orders. We also have our materialized view and this is how we're building out the silver layer as well. And then we have our gold layers here. We don't really need to get into these transformations. It really depends on what your data looks like, but this is just like a really simple high-level overview of what say a Python script that utilizes STP looks like.
This is also terrifying in the command line. I'm just going to keep repeating that, yeah. The whole thing about Spark declarative pipelines was the DAG for me personally. I'm just a simple person. I just like seeing the data flow. So if you're not a hardcore Vim user and you're a VS Code user, there are like open source uh like VS Code plugins that will allow you to visualize your DAGs which is really awesome. So this is a look at the like configuration file, really simple.
You can see that it just has our libraries and the Python files that we're including. We declare our catalog and our directories and things like that. And so we're going to see what it actually looks like to run this really quickly so that you have a good idea of what it looks like. So we have our pipeline code and this is just piping out the logic back to us so that we see it and then we can see these tables being made as well.
So nothing that I would get too in the weeds of, but if you just want to see what the output of these lines are, especially if you want to play around with it at home, it's very easy to just take a look, run it through and then see what the difference is. So in this pipeline really easily we're generating our silver and our gold tables through a really simple script and then we're showing that this verifies and works.
Okay then. So where does all the logging for this go? Like if you get a bunch of errors, you just have to search through this and hope that all of it's been saved. Spark does have a lot of logs. I think as a backup, it's really good practice just to keep these logs somewhere else safe especially if you're running so many different services that have their own logs as well. Logging ingestion and debugging is a very different part of running a platform that I think is a little bit out of our scope. , what do you think?
I think we maybe tried to cram too much into one thing one episode, but that's okay. >> wanted the entirety of Databricks into >> >> Okay, we could be busy. Uh good job we don't have a job anymore. Does it feel clear how you run this in a networked environment with high-powered clustered machines in a distributed system? Absolutely not. Yeah, totally. >> >> So I think where a lot of the complication comes from is really trying to run with multiple different clusters and jobs and the reason why we need clusters and jobs and all these different things at scale is because often times the more our business grows in this case as you guys raise whatever series you want to raise >> have a need for more performance, more machines, more raw compute and more reliability, right?
This idea of fault tolerance that, you know, you kind of need a job to complete no matter what. You need your data, you know, readily available no matter if, you know, there is a power outage in my case San Francisco or not etc. And so the way that we do that is by building infrastructure around it that keeps things running smoothly. Often times if you're then running with Spark declarative pipelines which and with Airflow at the same time, that gives you a lot of coverage in terms of actually scaling this out as you're having multiple containers running at once.
In each one of these we've only had one container running one service, but you're likely going to have like five to 10 Spark containers. All right, I'm just going to add to our diagram of scaling considerations. Uh what other things do we need for our scaling considerations? Um logging and being able to see what you're doing. >> healing for your containers. Orchestrating them. >> Jeez, Kubernetes and orchestration. Uh and then logging which sounds like it's expensive.

37:19 Scaling Diagram

I I was going to put the dollar signs on it on its own row. Okay, more dollars. >> >> Excellent for infra and our impending breakdowns as we realize that we can't do this. I would also add you're going to need more computers. More computers. Cool. It sounds like you guys have everything you need. Um >> >> always just get clone Lakehouse at home and that's like half of it. I feel like buried the lead there. already did the work.

38:17 Outro

Mhm. Yeah, I feel like as a as a local thing, like if it was just me wanting to play around and making my own free version, like is it doable? Yeah, it looks like I could do it, but I don't know if it'd be enjoyable. Like would would setting up your own open source version be more enjoyable than using Databricks free edition? No. >> not even close to being in comparison. This is like maybe Databricks uh circa five years ago, so to speak, in terms of like platform capability.
Um The features are of course all like there's a bunch of modern stuff that's that's recent, but like as a platform, this is like, you know, it's 50% of the way there. Yeah, it doesn't even have its own agent, you know? Like, I have to bring my own. Uh but, having said which, uh you do have a lot of the interop there as well. So, if you wanted to switch between I think it like is good if you want that flexibility, right?
Like, if you want to switch between, um you know, Spark or Trino or Ducky Be or whatever, it does give you the feasibility to be able to do that, which you probably wouldn't have if you were using uh Databricks free edition. Um But, yeah, I think we've got to go beg for our jobs back, don't you? Uh yeah. Yeah. Yeah. All right, then. >> be too late. >> this something that you'd want to give a go as a bit of a pet project?
Is this something that you would reasonably set up by yourself? Definitely remember to go and check out um repo. We'll put that in a link down below. Until then, we need to go get our jobs back, Bye-bye-bye. Bye. Bye. Say bye. Bye.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.