Databricks, the Good, the Bad and the Ugly

Summary

Honest review: Holly Smith, Staff Developer Advocate, shares candid opinions on Databricks features, highlighting three she loves, three with challenges, and three to avoid.
Product transparency: This video provides realistic perspectives on what Databricks does well, where improvements are needed, and what's actively being developed.
Summit navigation: Holly recommends specific talks to help attendees navigate over 700 sessions and find content most relevant to their needs.

Databricks, the Good, the Bad and the Ugly

Talk by Holly Smith, Staff Developer Advocate, Databricks
Databricks is the bestest platform ever where everything is perfect and nothing else could ever make it any better, right? …right? You and I know, this is not true. Don’t get me wrong, there are features that I absolutely love, but there are also some that require powering through the papercuts. And then there are those that I pretend don’t exist. I’ll be opening up to give my honest take on three of each category, why I do (or don’t) like them, and then telling you which talks to attend to find out more.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What are papercuts in Databricks?

Papercuts are features that require working through minor challenges or friction points before achieving the desired outcome, as opposed to completely broken features.

Why does Holly Smith share an honest product review?

As a Staff Developer Advocate who talks to demanding customers, Holly provides realistic perspectives on Databricks capabilities, limitations, and features under active development.

How does this help attendees at Data + AI Summit?

This video guides navigating over 700 talks, recommending which sessions to attend to learn more about specific features and solutions based on honest product feedback.

What's the scope of Holly's recommendations?

Holly covers features across Databricks products, sharing which ones deliver the best value, which require patience to use effectively, and which are worth skipping entirely.

Full transcript

`0:00` Welcome and introduction

Good morning everybody and welcome to data and AI summit. given the sort of circumstances. Um yeah, so welcome to this very early talk to uh basically talk about what else you should go and see at data and AI summit this year. There are over 700 talks this year and I hope this kind of whittleles down uh what's worth seeing. Uh for each talk I'm about to recommend there is a QR code to go with it. Um some of these will come up as already booked for context.
This talk was fully booked this morning. Um there are still seats. Uh so just make a note of it. you can stand in the standby line uh and then you'll be allowed to go in. Uh especially for some of the big talks as well, you might want to camp outside the room and when people inevitably don't show up uh you can get their slots. So I want to start uh by talking about the curse of the brickster datab bricks employee um or rather the curse of the customerf facing technical technical brickster. um the average person that we speak to um uses but isn't necessarily happy with our features.
If you think about it, the people who uh like the feature and can use them uh you never see them again. They kind of skip off into the sunset having a great time. Uh so by definition, the people that we spent our time talking to are the ones with very very high demands uh on our products uh and what they want it to do next. So something like this. Um, so I need to use features to use Azure backbone IPv6 on GCPC with nine lines of uptime.
Uh, and the data can never leave Switzerland. Um, so when it comes to knowing what realistically what data bricks does and doesn't do, we know. Um, so I hope this is going to give you like a very honest look into data bricks, what people like, where the opportunities for improvement are, uh, what's being actively developed on, and then you can go and learn more about that. I also do want to add a bit of a scoop disclaimer as well.
So, all of these things are technically public, but they might have been written in a very vague way. So, with that in mind, if you see something that you think is a bit of a scoop today, I'm going to ask very nicely that you don't post it on social media straight away, um, or I'll never be allowed to do a talk like this ever again. Uh, and if you also see anyone with a blue lanyard, that means that they are pressed.
And I would really like it if you could snitch anyone out. I don't think I can see anyone, so we should be fine. They shouldn't be in this room anyway. Uh, but yes. So, let's start off with some fighting words. Um, let's talk about one of my most hated things about data bricks, and that is uh community edition. Um, if you've never used community edition before, I don't blame you. Uh, show of hands if you've ever used community edition.

`2:59` Community edition updates

Yeah, no one's enthusiastically raising their hands, are they? Uh, so once upon a time, it used to be the way to start doing a personal project in data bricks. Um, but for some reason it was dep prioritized for year after year. Um, and it looks nothing like the full product at the moment. Um, I would call community edition kind of a secondass citizen. Uh, but let's be clear, it is so much worse than that. Uh and this is a real problem because people um are wanting to learn.
Um it's not just people who want to upskill on their resume. Um it's university students, it's hackathons, um it's training days. And let's be clear, it's better than it was. Um but there are still like a learning curve in some areas when it comes to using data bricks. Um and basically all of these problems uh just stop hitting yourself. So, wouldn't it be great hypothetically if we to put a project manager in charge of data bricks and tasked them with coming up with new and improved community edition?
Um, and that's exactly what we did. Um, Will Valori is the man for the job and he has spent the last few quarters um improving the experience dramatically on community edition. So, very briefly, what does this entail? So, the idea is that you should be able to try any data bricks feature. Um the Devril team, the team that I'm part of, we push quite hard to make sure that you can try everything. So for example, jobs, um if you're following like a tutorial from DB demos, uh if you're trying to recreate some training material, you should have the availability to use these things.
Um the limits for this week um are slightly lower, but I think they are going to have like a daily credit allowance rather than like a big chunk for a month. So if you I know get to the end of the day, you realize you've shot over, there's no kind of like slowdown in momentum, you can still sign on the next day and hopefully uh go for that. Uh and also we were able to use it um for our hackathon yesterday. Uh so it is already in use.
Um if you follow the right links, you can sort of get an early version of it at the moment. Um if you do have any questions about community edition, please go and ask Will. Not me. It's not my feature. Okay, let's talk about something good. Uh and that is of course Spark 4. 0. Uh this is the heroic effort that is happening as part of the Apache Spark community right now. Um I know this is down as good but I really wish I had a category for absolutely flipping great.

`5:21` Spark 4.0 highlights

Uh there is so much that is going on in this space right now. I counted either 10 talks um about this in aggregate or as a spark deep dive into features. I'm going to highlight some of my favorites uh in terms of features. If I didn't mention a particular area that you're interested in, please don't be discouraged. Uh so the first one up is oh that's not what I thought was going to be first up. Never mind. Uh native plotting uh with plotly.
Uh so at the moment or previously to Spark 4. 0, you had to mess around with different types of data frames. If you wanted to use pandas data frames for plotting, now you've got native plotting within uh Spark with plotly. We also have uh polymorphic Python UDTFs. That doesn't mean anything to you. uh these are UDFs that allow you to return a table as a uh as an output of your function rather than something like a column or a particular vector.
Uh we also have spark connector has had some massive improvements as well. Uh if you are involved in putting spark in places that don't normally have spark there is now a much thinner client in terms of the um languages that you can use. This has been dramatically improved as well with uh support for things like Go and Rust as well. Uh, and then finally, uh, my favorite section of this entire slide is streaming. There are some massive improvements that are going on here.
Uh, I would call them quality of life improvements, but I don't think that really scratches the surface of just how good these features are. U, we've got transform with state. If you've ever done stateful streaming, you've come into flatmap group with state and you've just wanted to cry because it's so difficult. Transform with state is the new version of that which is a lot easier to use. Uh we've also got things like improved checkpointing so you'll get better performance.
Um there's also things like uh improvements with the state store not just in terms of the improvements uh of the speed but the interoperability as well. Um so with that in mind I'd whittle this down to three talks that I would recommend. Uh and it ended up being these three. If I've just shown you all those things on the screen and you're like I don't know what any of those things are. I really recommend having the intro talk be the one on the left which is uh what's new in Spark from and Wenchan.
That is on Thursday at 12:30. Um if you are interested in some of these things we've got some very niche specific questions there is an AMA that and DB are doing later in the week. Uh but if you can't make that for some reason um there is also a Devril booth that is downstairs in the expo hall at the moment. We've got a stand dedicated to Spark full of Spark experts. So, if you do have particular questions about what's coming out in 4.
0, then go ahead and have a look. If you are quite into the weeds with Spark, um the DSP2 stuff is fascinating with a very wide reach and impact. Um and it has been a real uh community effort to make that happen. So, if you are consider yourself kind of a super spark expert, um I really recommend talk uh for DSV2 in Spark. Well, that's enough positivity. Let's go back to trashing things. um you know what was coming uh is the Marmite of data bricks products um and that is Delta live tables uh the people who love it put it in everything uh and the people who don't like it are not quiet uh and I say this as someone who has an oil painting of Marmite uh in their house at home uh again some teams love it uh the number is going up over time of DT users not Marmite eaters I have no data on Marmite god I sound like the crazy momite my lady, don't I?

`8:34` Delta live tables overview

Um, I kind of am. Uh, in general, um, people who have to chan through astronomical amounts of pipelines, if they're doing like a migration and they can't spend time curating every single pipeline, DT is an excellent option for you. So, whenever I speak to kind of consultants or people that do this kind of big lift and shifts, they love Delta live tables. Um, sorry, quick question. Who here has ever been part of a migration and migrating things to data bricks?
Okay, now keep your hand up if you didn't run out of time and everything was to the standard you expected. Oh, literally no one. Okay, Delta Live Tables is an excellent candidate for that kind of thing where you don't have the time to curate and perfect everything. Okay. Uh what else do uh what else do people say about Delta live tables? Um, so if you need something that's kind of good enough, like if I'm doing something, I don't want to waste time doing it from scratch.
You know, I could bake bread for my mommite every day, but I don't. I get it from the shop. Uh, and if you are still dead set on detailed configs, then uh maybe uh this is not for you. Sometimes a pipeline does need that extra level of config. Although, as data bricks get simpler and it does start being able to kind of optimize jobs in the background, this is becoming less and less true. We are seeing a little bit of a rise of people overwriting the default configs and just kind of making the performance a lot worse.
So, I'm hoping that is going down over time with the help of some other features that going on as well. Uh, but in a cruel twist, it's also not There we go. This is the Spark busy board for people who are uh if you find yourself having to twiddle with absolutely everything and you're making your performance worse, I'm going to get your Spark config busy board like this. Um but yes in a cruel twist um delta live tables is not quite simple enough for people um who are maybe coming at it from an analytics perspective and so that can make it a little bit challenging.
So hypothetically maybe wouldn't it be prudent perhaps to maybe invest significant engineering time into this and then if you had hypothetically spent a lot of engineering time improving it uh wouldn't you then update any big announcements of the biggest conference of the year uh and so that's where I'm going to go and recommend this talk uh by Michael Michael Armbrust and uh from engineering at data bricks um declarative pipelines simplifying data engineering workloads Again, I don't want to give too away too many spoilers about what's going on here, but I really really recommend this talk to get the latest updates uh on what's going on there.
There is also an AMA from on Thursday at 2 p. m. Um, so if you have any particular tricky questions, um, please go and herang him. Oh, what is going on here? Right. Okay. Um so I want to talk about something that is now good. Um and that is data bricks on datab bricks. We um aggressively dog food our product at datab bricks. And I have put this one in good because from your perspective uh this is really good. Um but unfortunately the intense nature of the dog fooding means that it's not necessarily the most simple of processes in internally.

`12:00` Internal dogfooding projects

Uh and so we do enforce this dog fooding and we do get excellent feedback about where to improve. But the part that goes unsaid is that kind of like feedback process. Uh I don't know if you oh uh know at data bricks but we have a very kind of like open and honest kind of culture which means that any of this feedback does tend to be fairly quick and fairly fullon. So I do want to recommend two talks for you today um about how we've done these data bricks internal projects where we get people at data bicks to use data bricks.
I know some of them have probably come up already that maybe they might be full. These ones are surprisingly uh not well highlighted at the moment and I do really recommend that you go and do these. So the first one talk on the left um is about our field AI assistant and that's built on the data bricks agent framework. It scrapes data from all over the place. So things like our CRM, our Google Drive. Um, but it also helps you do stuff like uh prep for your meetings and your proposals and draft things like outbound emails as well.
So it's not going to replace your sales team. Let's not be silly. Um, but it is going to make things go faster and it is going to help prevent kind of dud meetings and dudcoms going out. The second one is when we were realizing we were spending a lot of money on competitor BI reporting tools despite us having very similar capabilities. Uh and that's how we forced hundreds of people at data bicks to use data bricks for the first time.
So in the long run uh you know this is good but for the team I imagine there were fairly some fairly ugly moments in that process. Um and yes these are both very good technical talks. Um but if you're also improved in kind of um involved in improving processes uh within teams this is also get good from like an organizational behavior kind of perspective to attend these talks as well. Okay. Um so the next thing I would ask you put it out to the audience to say uh what you don't like about data bricks.

`14:26` MLOps and developer feedback

Uh but fortunately has already beaten you to this uh by telling us quite publicly what she doesn't like. uh don't worry despite all this she does still like the platform but yeah she's not a big fan of the developer experience uh but she is going to talk later in the week about her experience with MLOps in data bricks um and how she is able to kind of turn the platform into the way that she thinks it should work uh I do really recommend going and having a look at her talk if this is something that you have been struggling with all right let's head back to some positivity uh I will say I I have been blown away by the datab bricks assistant.
Um, and this has been my guilty pleasure for quite some time. I make a lot of demos with features that I might be unfamiliar with. Uh, and this allows me to power through um at speed that I never would have thought is possible, candidly. So, getting started is a lot easier when it's just something that you kind of know, but you've kind of forgotten, but you'll know it when you see it. Uh it's also fantastic for understanding my co-workers code given that they're all more talented than me.

`15:04` Databricks assistant

Uh I can now learn what they've done rather than teaching their code as like a no touching zone. Um the contextual ele um element is also very good as well. Um there are times where I've been doing demos but it starts giving spoilers to the demos because the recommendations are just so good. Something that is a little bit kind of hit and miss is the error handling. Uh, most of the time it's okay. Sometimes it does need kind of two or three times to mash it to kind of bully it into the correct answer.
Um, and as much as I'm dunking on this, I do occasionally work on other platforms, uh, which don't have this button to kind of like fix your code if an error comes up. And I must say, when that button is gone, I really really miss it. Uh, but the biggest miss for me, um, is unfortunately it's not great when it comes to brand new features. And as someone who's using it generally in the very very early stages of a brand new feature coming out, it tends not to be too great at kind of learning about these things.
But with all that in mind, I really do recommend that you go see uh Gal and talk about the assistant and what's in there. I have seen this talk before. Originally, I saw it on the agenda. I thought it was going to be a little bit. I hope they're not offended or they don't watch this talk. Um, but ultimately I thought it was going to be I don't know, a little bit simple, like how to click a button, how hard can it be?
I realized I was missing out on so many of the features of the assistant. Uh, there were many things that I didn't realize you could do. I do recommend that you go uh and have a look and you can learn about all the things that it's doing. So, for example, um, it can now optimize your code as well, which is pretty impressive. Again, is it going to give you the perfect configs in the world? No. but it can point out those things that are a little bit silly that you might have done that you might not have realized and will help you optimize your code and again all it is just mashing a button.
So I will recommend heading to that one. Okay then uh next one very ugly datab bricks runtime upgrades. Uh can I actually do a bit of an audience poll at this moment? Um, for those of you who have done a data bricks runtime upgrade, uh, can I get a hand up if it took you more than a year to get that done? Yeah, we've got a few and more than nine months. No one. Okay. More than six. All right. Three months or around that amount.

`17:32` Runtime upgrade challenges

Okay. Not very many hands have gone up. So, hands up if you've never done this before and everything's still on like DBR 7. 5. Okay. Um I can see from your expressions here we're not re we're like we are on the same page like uh it's an absolute faf. Uh it takes time away from other projects and it's also annoying to justify from an ROI perspective. You then have to kind of like explain to upper management yes I want to upgrade.
Especially for those of you who are taking more than a year um to be able to upgrade. You really do need like a big investment to get this done. So what if I told you there was a new way to manage runtime that took less effort and was less errorprone. Um spoilers it does involve you using serless. Um but there is so much clever work going on underneath to do with environment manage to do with issue remediation to do with error handling that give you the latest features with the least amount of work.
Uh and so that I'm going to recommend this talk uh who is the brains child behind it. uh a breaking up with spark version. So if you put your hand up and it takes nine months or more, I really recommend giving this a look. Will this solve every single thing? No. Will it make that process significantly quicker? Absolutely yes. Now my next talk has got absolutely nothing to do with data bricks at all. I just wanted to give it a mention uh because I thought it would be funny.

`18:56` LLMs, comedy and table formats

Um, and that is professional funny lady. Uh, comedian Erin Staples is joining us this week to talk about LLMs and comedy. As someone who believes that every aspect of life should contain a little bit of comedy, I agree that there is now an opportunity to scale this with LLMs. Um, but as comedy is so subjective, uh, this make eelss quite tricky and that's what she's going to be talking about. I can't actually attend this one and I'm gutted. out of all the talks that I'm going to sit and watch at home by myself later, uh, this is probably going to be high up on the list for me.

`19:57` Customer stories and resources

Okay. Um, bad table format wars. I have to be a little bit careful what I say here because there is an enormous amount of work that has gone on in the open source communities and I do not want to denigrate that kind of work in any way, shape or form. um especially as there's so many people who are like contributing graciously with their time to these projects. But from a customer's perspective, it looks something uh like this.
So step one uh is pick your tribe. Um uh you pick the one that the sales guy told you probably uh you just hope that you don't really need the other one. Um then you give out and go with Excel. Wait, no, no, no, no, no, no. Uh but let's be really honest. of all the kind of table format projects that are going on, they are trying to solve very similar problems. The feature requests that are coming from their users look very similar and over time these two projects or three projects I should say are getting closer and closer together.
Uh and so to talk about that, who better um than some of our extabular guys who joined us at data bricks this time last year uh for the future of open table formats delta iceberg and more. So, I really recommend having a look at this one. Again, this one's probably going to be one that might be booked out quite quickly, but this one is definitely worth camping out for. Okay. Uh, now on to some good things. And I couldn't put a QR code here for each of these as there are so many customer stories, but I went through pretty much all 700 talks uh, and I had a look at some of the names that I know are speaking.
I know the work that they're doing is either very interesting. It's either a novel approach to using data bricks. Um if you uh are working in a particular industry and you see some names up here that you think are, you know, kind of relevant to what you're doing, then give it a go. Uh but I would also encourage you if you don't have like, you know, very specific industry needs, um do also go and have a look at some other things that maybe outside of your industry.
Obviously, if you've got to deal with the kind of like high privacy data, you know, there are other ways to see that, but you know, I would encourage you to have a look. um other industries this week as well. Uh I've done something like 40 customer data bricks projects so far. And once you're like ripping up and looking into the guts of data bricks and how the platform works, it tends to not necessarily matter too much um about the industry specific.
So um or also for every single advert there also should be a talk associated with it. So as you're going around um you should either be able to find a talk or maybe a blog for them as well. Okay. And the next one is kind of awkward because it's redacted. Uh I can't tell you how good this session is going to be, but this obviously up here for a reason. Um and here is the abstract for the session. In this session, you will learn about how data engineers can build intelligent applications on data bricks with production level concurrency and simplified operations.

`22:42` Mystery session and keynotes

Uh, we'll walk through reference architectures and example use cases while unlocking new possibilities of AI and analytics. How clandestine. What intrigue. If only someone knew what this was about. So, sign up now. Uh, this is pretty ded at the moment because it's incredibly vague and no one really knows what this talk is. Uh, I highly recommend you scan this and you sign up for this talk. Um, I have a suspicion, maybe a mere hunch, um, that later this week, this is going to be one of the most popular sessions of the entire week.
So, you've got the scoop, sign up, make sure that you can attend. Uh, speaking of things like that, we've also got ultimately the best thing and that is, you know, the keynotes. I know what we're announcing this year. They are definitely worth attending. I appreciate they are like two, three hours in the morning and that can be quite big. Um, if you are someone who cannot sit still for three hours, I feel you. I've got that as well.

`23:48` Closing and podcast plug

Um, then I recommend like dialing in. There's a virtual experience. You can dial in with your phone and you can watch that remotely if you wanted to sit in your hotel room and watch it rather than sit in the room for 3 hours. But if you do come, the atmosphere is absolutely electric. I do recommend it. Uh, and again, the announcements are just absolutely phenomenal this year. And of course, finally to cap us off today, the absolute peak of data bricks is of course uh the data bricks podcast that I run.
Uh this is the podcast where and I take our favorite features from the last 30 days and we try to shoehorn them into a single architecture. Is this realistic? No. Will you learn something? Maybe. Um, and we are doing a live recording uh including many of the features from the open source communities uh from Spark, Delta and ML Flow. It'll be fun and you might even learn something. Okay, that's it for me. Thank you very much for coming to this talk.
Um, I hope you have a wonderful time at data and AI summit and I'll see you around.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.