Databricks Enterprise AI: Building Teams, Culture, and Data Foundations

Summary

Arsalan Tavakoli-Shiraji, Databricks co-founder, describes how field engineering teams use AI to automate research, accelerate customer advisory work, and deliver complete end-to-end pilots rather than demonstrations alone.
Maintaining strict hiring standards and a culture of transparent communication, with clear strategy alignment across the organization, is identified as critical for scaling from 200 to 30,000 people without losing execution quality.
Data transformation through Unity Catalog and governance frameworks must precede AI initiatives, because AI amplifies whatever is in the data — including its flaws — making a governed data foundation the prerequisite for trustworthy AI outcomes.

Databricks Enterprise AI: Building Teams, Culture, and Data Foundations

Rapid AI transformation requires more than technology. In this fireside chat, Databricks co-founder Arsalan Tavakoli-Shiraji shares how enterprises scale their teams, maintain culture, and establish data foundations essential for AI success. Drawing from 13 years of building one of tech's most influential companies, he reveals the organizational and strategic principles that enable rapid execution at scale.
Discover how forward-deployed engineers enable faster POC and pilot delivery by combining Databricks capabilities with business problem-solving, why maintaining strict hiring standards is critical during company scaling, and how transparent communication and clear strategy alignment drive organizational execution. Learn why data transformation through Unity Catalog and governance frameworks must precede AI initiatives, how the cloud-first approach differentiated Databricks early on, and why open components like Delta enabled customer success rather than vendor lock-in.
🤝

Chapters

FAQs

What is the role of forward-deployed engineers at Databricks?

Forward-deployed engineers combine deep Databricks technical expertise with business problem-solving to partner with customers throughout their journey, from initial evaluation through production deployment and enablement. AI has accelerated their work by automating research tasks and enabling the team to build complete end-to-end pilots rather than just point demonstrations.

Why does Arsalan Tavakoli-Shiraji stress maintaining hiring standards during rapid growth?

Maintaining hiring excellence is identified as one of the critical challenges companies face when scaling quickly, as the temptation to lower the bar to fill headcount faster can dilute the quality and culture that made the organization successful. Arsalan frames this as one of the leadership principles that has guided Databricks through its growth from 200 to approximately 30,000 people.

Why must data governance and Unity Catalog come before an organization's AI initiatives?

AI systems built on top of inconsistent, ungoverned data will produce unreliable results regardless of model quality, because AI amplifies whatever is in the underlying data — including its errors and inconsistencies. Arsalan argues organizations must first establish clean, governed data foundations through Unity Catalog before layering AI on top.

How did open components like Delta Lake differentiate Databricks from competitors early on?

By building on open formats like Delta Lake and contributing them to the open-source community, Databricks enabled customer success without creating vendor lock-in, which built a level of trust that proprietary black-box solutions could not match. Arsalan cites this cloud-first, open-components approach as a key strategic decision in Databricks' early differentiation in the market.

Full transcript

[00:18] All right. Bro, welcome back to Summit Live. As you can see, this is live. I'm Holly Smith and I'm here with Ari Kaplan. How you doing? Great. Welcome back for everyone tuning in 160 countries around the world. And we have the distinguished guest Arsalan here. Welcome. Welcome.
[00:33] Thanks for having me. One of my favorite parts of the Summit always. Uh, it's a pleasure to have you. And first, tell me about the field engineering department. What what's that? What exactly do we do? Yeah, exactly. What is it what do you team do? Did you just make it up? Um, look, I think that the easiest way to think about it is field engineering is
[00:51] everybody to anybody technical at Databricks outside of the core R&D org, um, you know, in many ways. And the way that I would think about it is we partner with the sales side of the house. And our main goal is to help customers go along their journey everywhere from how do I understand what
[01:06] Databricks actually is through how do I basically like evaluate it to how do I actually get it to go live, including training and enabling, um, you know, the organization and then figuring out what are the next things that I have to adopt. So, it's super fun because you basically get to spend a bunch of time
[01:22] with customers and also one foot in the product as well. So, you're in a really unique position because you are so well situated to start using AI. And if there's something that you don't quite like, look, the engineering team is not that far away and you can say, "Hey team, come on, get it together." How has AI affected what
[01:38] your team has been doing and how they work? Um, so it's affected it a lot, right? And I'll pick, uh, three maybe examples that I think are fair. Um, one is often times in field engineering you're like the trusted advisor to the customer, frankly the trusted advisor often times
[01:55] to the sales rep and the partner as well. and they have tons and tons of questions. And often times those questions require you to be a walking encyclopedia. How do I go research it? How do I go figure out what the right answer is? And now with AI, it basically can do a lot of that research for you, validate it. So, it's sped up and kind
[02:10] of helped automate a lot of time you spent there. Second, you spend a lot of time building for customers, right? So, whether that's I want to build you a demo or help you basically a POC, and customers are now also sitting and saying, "I don't just want a demo. I want you to build me an end-to-end pilot and see what that looks
[02:25] like." And something that previously could have taken weeks to build, you can now do in hours or days, so you can basically do many more. And then lastly, uh I think everybody in field engineering, you know, loves to learn, but you're used to doing what I call push-based enablement, where you sit in
[02:42] the class and you go through all the sessions. And as we saw from the announcements this morning, there's just way, way too many things, and you kind of need that content only when you actually need it. So, AI has a lot allowed us to move to this pull-based, um you know, enablement, where I'm going into a customer, I know this is what I
[02:58] care about, how do I help learn, what are the products that they care about, the topics, so I can deliver it to them. So, it's that just-in-time learning. Those have all been key to helping us continue to scale. Yeah. And so, one thing that I have seen is a bit of a shift. So, for context, I used to be in Arslan's organization. I left not for Arslan, for other reasons.
[03:15] That's what you tell me, right? But, Uh but, I worked in a what was our kind of consulting business, but that's really been turned on its head, and we now have forward deployed engineers, FTEs. Yes. What are they, and why are they popular now? Why does everyone have them? I think certain things become cool,
[03:31] right? So, it doesn't matter, like, whatever the question is, the answer is FTEs will solve them. But, look, I I think that the shift that's happened is previously, um when you looked inside of, like, professional services, um and implementation and delivery, it was a lot of how do I set up this platform,
[03:49] right? Like, how do I I from Teradata, or how do I migrate from kind of EMR or something like that? Um, and now what's happened is as the platform has gotten broader in the AI era, it's shifted to a little bit more of can you help me solve this business problem, right? Um,
[04:05] and so I think it's kind of given a new name to something that we've been doing for um, you know, for a while candidly, but the nice part about Databricks now is it has everything from okay, I do the ingest in the platform including AI, including applications as well. And so customers are asking us to come in and
[04:21] say, can you help me redesign this process from the ground up for being AI? Can you help me get there a lot faster and make the the right decisions? So we've been excited to partner with them and the main things that I think differentiate us from the rest that we push is a lot of times um,
[04:37] AI, I mean FDEs tend to be what I call the trust me bro model where whatever your problem is, I'll just solve it. I'm like, I don't know what that means. You need to know what success is. You need to define it. You don't want to outsource something to just FDEs. Um, so you don't want to get they they build you something like it's a black box. So
[04:53] like when we build it is we get them there faster for their business problem, but with all the open components of Databricks so then they can take over control and they can continue evolving it without some kind of high exorbitant fees that often times go with some of these FDE engagements. Yeah, I hear yeah, a lot of positive
[05:08] things about FDEs. That's like the way way to grow as we grow. And I wanted to hear like you you've been here from the beginning and then you have to have certain, you know, uh, quality of employees and collaboration as you grow through all the different
[05:23] phases of Databricks and we've been doubling every so many years. Like how do you keep and maintain that like the highest caliber quality of staff and communication as we grow? I think the short answer is you put a lot of effort into it. You make it a priority, right? I think that there's a
[05:40] there's a couple of things about Databricks um, and it's easy to say it, but it was really really important which is one, culture. I feel like at some places culture is just something that they're like, this is what happens. You go into it. Or we have some leadership principles on a wall somewhere and you know, hopefully
[05:55] but here it's like those principles you basically are super thoughtful. You've seen this kind of evolve them over the last 13 years and they factor into everything about who we recognize, you know, who we promote, who we hire and then we also make it a really, really
[06:11] big deal of spending time on hiring, right? So we're constantly looking at are we evolving our hiring process? Are we making sure that we're deliberate about it? And what we always say is we much rather have a false negative than a false positive cuz the cost of a false
[06:28] positive is high and everybody will tell you, hey, when we were going from 10 to 20, you can't keep hiring like that from 20 to 50, from 50 to 100, right? And now Data Bricks is over 10,000 people and field engineering is I think at last count you know, well over 3,000 and so you
[06:43] still want to put the same care and deliberation on it today. That's the only way you can get there. If you don't make it a critical priority, it just degrades gradually over time till one day you wake up and you're kind of a crappy big company. Yeah, and I've seen a lot of great validation from Glassdoor to like all of
[06:59] the culture surveys uh rating Data Bricks as one of the best places to work in all different angles and well, the feedback I got I was just at the analyst relation and they're pulling me aside saying, you know, the transparency from the executives um to all the employees is is just like
[07:16] so refreshing and so unique. Just people being able to say um professionally what's on their mind and not hiding things back helps the company progress. I think I think that that's true cuz what I found is uh again, it sounds kind of cliche but
[07:31] everybody people appreciate two things. They one, they appreciate what it's not that whether something is good or bad, they appreciate transparency to your point, right? Cuz it's like I want to And then the second is when you ask me to go do something and you give me the
[07:46] the what, I'll figure out the how, but I need to know the why. Yeah. Cuz the why helps me get behind it. Because in general, most of the times you want to believe that people are like when they're pushing you somewhere, there's a logical reason for why. And we've had a lot of things that have happened over the history of Databricks,
[08:02] for example, like deciding to never go, um you know, on prem in basically like a pure on prem offering and being kind of cloud first. There was like open sourcing things like Delta and many of our capabilities that people were like, "Why?" But when they understood the why and here's the strategy behind it, they were able to get behind it. And
[08:18] fortunately, uh we've had more things be right than wrong, I would say, uh you know, up to this point. on prem thing early on was a big deal with our investors coming in, but you and Ali stuck to it and made it such a big iconic company. So, you've been
[08:33] um you know, very very early days, you've seen the whole company grow. You've still stayed very much in touch with kind of like founder roots and everything like that. People like to give a lot of advice to founders and people growing businesses, especially with the whole AI stuff going on. Is there any advice that you see over and
[08:49] over again that you think, "Oh, wouldn't have done it like that. That's terrible advice." Uh I mean, there's a lot of them. How much time do we have? As much as you want, Ali. Biggest eye rolls. I think that there is a couple. One was on the hiring. What people say is one of my favorite is like, "Look, you can't
[09:06] just hire basically like A+ people all the time. So, if you hire A people and B uh B people, the A people will train the B people and develop them into A people." And I was like, "No, that's not what happens." When you do that, the A people are like, "Screw it. I'm out." And then the B
[09:22] people get insecure and they hire C people to make themselves look better and then you have this kind of gradual decline. So, I think that that's one like that happens. It's that Frankly, I would tell um you know, a bunch of leaders, "If you yourself, especially in the early days, are not spending nearly half of your time recruiting and hiring,
[09:39] you're going to basically regret it down the line. You're not setting yourself up for success. I think that the second one is you basically talked about this. Right now, everybody's AI pilled. If you're not AI all day long, if you're not doing the If you're not using AI and so I'm like And so I'm like, guys, AI is
[09:54] a thing. It's a mechanism. If the answer is like, if you're not achieving X, then AI may very well be the way that you get there, but otherwise if you just like a lot of people are running frenetically worrying about like, am I using AI? I'm more AI pilled than you are and I have no idea what that even means anymore at this point. Very wise. That's something everyone
[10:10] should take into account. Yeah. Um yeah, so that's been fascinating. So, um I'd love to hear more of your perspective on I you know, you would you've been 10 years, 12 years? 13. 13 years. Wow. Don't shortchange me.
[10:26] 13 years has aged me at this point, you know, so You look just over 60, right. Oh, there we go. Different jokes. Um so I guess when you were very early days, I mean are there things that you look around that you never could have imagined in your wildest dreams? Or like some of the things that make you like trip you up and think, wow, we really
[10:41] have grown. I mean there's a lot it's funny. Um we oftentimes get asked this question that says, did you guys ever imagine X? Um and the short answer is no. We spent most of the early days trying to figure out how not to die as a company, right? What?
[10:57] But yeah, right? It's very funny. All of the people that you were worried about that were going to crush you, nobody has ever even heard of them anymore at this point, um ironically. Um but look, you don't really get a break because uh you know, the one thing is, you know, when new people join Databricks, they
[11:12] ask me it's like, hey, can you let me know when will it stop feeling like I'm just drinking from frenetic fire hose? And I was like, not sure. It hasn't happened in 13 years. I'll let you know when I get there. Um but there's certain times where like this morning when I came to Moscone and you look around and
[11:27] there's lines wrapped around the building from five different directions. They're talking about over 30,000 people in person and how many online. And I still remember the first uh Databricks Summit. It was called Spark Summit way back then. I think we were
[11:43] like 200 people in December of 2013. And at that point, our goal was, "Can you please just at least say the word Spark somewhere in there?" You can 95% OF THE TIME YOU can talk about Hadoop and what it's doing and nothing in product. And now you're
[11:59] talking about some of the world's like most iconic companies and some of the greatest kind of like startups are sitting there frankly talking about, "Here's how we've transformed the business." And so, there's times when you look at that, you're like, "Wow, Yeah. that is definitely different than, you know, what I thought it would be. But at the same time, you try not to let
[12:15] yourself get caught up in that because honestly, there's so much more to do and you want to keep that sense of urgency, that frenetic pace, that setup making things happen and move quickly, which I think is really important. Yeah, I think that is a real key part of Databricks culture as well is this idea
[12:30] of um just keep executing, just keep going. Uh I do remember one of our CKO's uh it was all about kind of delivering. Yeah. I think Ali wanted execute written across the stage and we had to point out why that's really bad optics. You can't have the word execute in big letters on
[12:46] his stage. Yeah. Um but we've got maybe time for one little tiny question. But I mean, you know, you've got to speak to a lot of customers this week. What do you think is going to be the most common piece of advice that you give people? Uh so, I I've got to only pick one. You're just
[13:01] basically trying to pigeonhole me into this. Um I think the one thing that we tell people is everybody's excited about going on an AI transformation. And Ali's heard me say this before that they're all screaming the AI part, but very quickly you I think if you saw the keynote this morning um where from Pepsi
[13:17] they said like, "An AI transformation is a data transformation, right?" And I think that that's a So, I think a lot of people we spend a lot of time talking about in in to get it right. Genie ontology basically uh Unity catalog, Unity AI gateway, all of those governance frameworks are the things
[13:32] that let you set the proper foundation that then let you go to all of the exciting parts afterwards and be able to run as fast. Without those, you'll be in trouble and I think I've seen already all my conversations with customers, they're really excited about that and figuring out how do we set the right foundation for it.
[13:48] All right, wonderful. Thank you so much Aslam for spending some of your precious time talking to us today. Thank you so much. We are going to head over to some videos about some training for you to see what's been going on at Summit. Thank you. Thank you. Thanks, guys.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.