Data + AI Summit Keynote Day 1

Summary

Free and accessible learning: Databricks Free Edition lets anyone learn and explore data and AI for free using the same Databricks Data and AI platform used by millions of professionals.
Governed app development: Databricks Apps announced general availability, enabling secure and governed data intelligence applications built with traditional code or visual no-code approaches.
Intelligent automation: Agent Bricks auto-optimizes AI agents using your data, MLflow 3.0 unifies experimentation and observability, and Model Context Protocol brings standards to the Databricks Data and AI platform.

Data + AI Summit Keynote Day 1

Watch the full keynote from day 1 of Data + AI Summit 2025
Chapters:
Presenters:
Ali Ghodsi, Co-Founder and CEO, Databricks
Reynold Xin, Co-Founder and Chief Architect, Databricks
Hanlin Tang, CTO, Neural Networks, Databricks
Nikita Shamgunov, CEO, Neon
Justin DeBrabant, Director, Product Management, Databricks
Kasey Uhlenhuth, Director, Product, Mosaic AI Platform, Databricks
Holly Smith, Staff Developer Advocate, Databricks
Dario Amodei, Co-Founder and CEO, Anthropic
Greg Ulrich, Chief AI & Data Officer, Mastercard
Jamie Dimon, Chairman and CEO, JPMorgan Chase
Richard Masters, VP Data & AI, Virgin Atlantic Airways

Chapters

FAQs

What is Databricks Free Edition and who can use it?

Databricks Free Edition allows anyone to learn and explore data and AI for free on the same Databricks Data and AI platform used by millions of professionals, lowering barriers to entry.

What is Lakebase and how does it support data apps and AI agents?

Lakebase is a fully-managed PostgreSQL offering announced in public preview, purpose-built for powering data applications and AI agents with enterprise-grade reliability and Databricks governance.

What are Databricks Apps and what can they do?

Databricks Apps, now generally available, enable teams to build secure and governed data intelligence applications using both traditional development and visual no-code builders like the Genie App Builder.

What is Agent Bricks and how does it help developers?

Agent Bricks automatically optimizes AI agents using your organization's data, simplifying agent development and deployment on the Databricks Data and AI platform.

Full transcript

`0:00` <Untitled Chapter 1>

Please welcome to the stage datab bricks co-founder and CEO. Awesome. I actually thought we start with a little bit of a look back. Believe it or not, this picture is from 10 years ago here at the Moscone Center. We were so excited. We had over 3,000 people in the audience. We were like blown away. We're like, "This is impossible." Uh, and back then we were doing AI, but really our focus was on data engineering, making big data simple.
That was the big focus. And we had just released Apache Spark a few years earlier. It was just a 1. 0 release. And it was a big deal for us. And then three years later, this is a picture from three years ago. Um that's again Moscone Center, but now we actually had over 5,000 people. Okay? And we thought this is the maximum we're going to ever do. Uh but now we have started focusing more and more on artificial intelligence.
We actually changed the name of the conference from Sparks Summit to Spark Plus AI Summit. And um I'm sure none of you remember, but I actually went back and looked. Uh I was on stage and I actually live coded and there was no vibe coding, no assistance, so it's actually real code um with TensorFlow generative AI and I made a picture of Boris Johnson back then. And with that our focus started uh expanding from data to both data and AI.
And if you fast forward actually till today, it's amazing to see where this conference and this community has come. We now have actually over 22,000 people here at the Moscone Center. We've Yes. Super exciting. It's actually a citywide conference right now. We've taken over all of Muscone. It's basically four blocks and it is the largest data and AI conference on the planet. 65,000 people worldwide are watching right now.
And the stat that always blows me away is that there's 150 countries represented. Okay. Uh so this is not like a just US or American or west coast centric event. It's under 50 countries. Okay. Uh we have super awesome program. 700 sessions and trainings happening over 200 or you know 350 customer teams are presenting. So that's you guys uh are going to present the projects that you've been working on. It's going to be super exciting.
And we also have 180 exhibitors. And you know the backbone of this conference over the years is open source. So the open source projects that were launched in this conference and that you all built now have over two billion downloads just on Apache Spark alone. Over a billion downloads on Delta Lake. Uh iceberg which we adopted last year has over 360 million downloads. And then MLflow, which we use for all of our ML ops and all our experimentation for generative AI, has over 300 million downloads.
And together, this is why our mission at data bricks is to democratize data and AI. We want to bring data and AI to as many people on the planet as possible. And I want to take a second to thank our sponsors who make this possible. Um, especially our legend sponsors, the hyperscalers, AWS, Google, Microsoft. Yes, thank you. and also the gsis Accenture and Deote and all the other ones that are on this slide. And I also want to send a special thank you to all the other partners we have.
We have the system integrators that help us actually build all this with services in your companies. Uh we have the technology partners that are super essential. They augment the platform. We integrate with them and they're the ones that actually make us do things that the platform alone cannot do. the built-on partners that built on top of data bricks uh their solutions and they sort of innovate and push the boundaries of what you can do.
Uh and then finally the one I'm very excited about and which is very important are the data providers right those are the ones that you can get on the marketplace on data bricks you can do delta sharing you can share the data set with other uh BUS and other organizations in the ecosystem and together with all of these partners the most amazing things are our 15,000 customers that's all of you who are in the room that's the real impact that's where the interesting things are happening so thank you All.
So, we have a super exciting program. We have actually announcements both today and tomorrow. So, you know, some people think, "Oh, the announcements are the first day." No, there's going to be new announcements tomorrow, too. So, you know, if you're booking flights to go home tonight, extend them, please stay another day. Okay? We have very exciting things tomorrow. Uh, I thought I'd just start off uh by sharing uh a video on one of our customers and what they're doing that I think it's really cool.
So let's watch. fly like an air taxi. We're generating gigabytes a minute of data on our airplane. We want to push this data to our developers, engineers so that they can gain insights from it very quickly. So datab bricks is the platform that we have chosen to make this data democratization happen. Using data bricks has allowed us to quickly develop some of the sandbox. We leverage the endpoints. We leverage the model serving.
We leverage the conditional models. So all these technologies that datab bricks has provided have proven to be very key for us. What we've tried to do is infuse the best of the way aviation does things with kind of the best of Silicon Valley. videos back there as well. If you want to go have a seat, it's easier to watch. Okay, so you heard and what they're doing. You're going to hear from hundreds of hundreds of customers here at this event what they're doing with data and AI.
It's super impressive. But the reality on the ground is that it's actually really hard still to succeed with data and AI. U especially if you're in a company that has been around more than say 10 years or 15 years, chances are that your architectural diagram of your company's data and AI infrastructure uh looks something like what I have on this slide. In fact, this is probably a simplification. Okay, you at least have one of each of those services that I'm putting on this slide.
You probably have it from some vendors or maybe you're using some hyperscalers services for this. You probably have several data warehouses, maybe some onrem, some in the cloud. You have a bunch of unstructured data. They're stored in the lakes. Um, you have, you know, real time solutions that you've custom built for your organization. You have lots of ETL, data engineering, data movement that's happening that's typically bespoke.
It's been built over the years. super hard to move. You don't want to break anything with that. Uh and then of course all the organizations we work with, they have one of each of the BI tools that exist. Like they have literally when I ask which BI tool do you use? They always say well we have every one of them. Okay, dating back 20 years. Um and then of course recent years there's been a lot of focus on data science, machine learning and now generative AI.
So you see all of this in your organization and what that means is that this is creating a complexity in most companies. It's slowing down the organizations. The costs are super high and it's creating a proprietary lock in. So at data bricks since 5 10 years back we think this is the biggest problem. Actually if I look back at our sort of history we came from academia. We thought we were going to change the world overnight.
And the biggest thing that we learned in the last 10-15 years is what we see on this slide. This is why everything moves slow in the organizations. So that's why we about 5 10 years ago set out to simplify this architecture and help organizations move faster. That's what we really wanted to do. We sort of brainstorming how can we simplify this? What can we do? And what we realized over time is that the reason this is a really bad architectural sort of situation is that each of these things has its own data sitting in it.
And that's what's really locking people in. There's data and there's metadata. You see that with that binary icon that I have. Uh and then not only that, and this one I think we learned a little bit later, each of them have their own security model, their own access control, uh their own governance because you know, everybody's getting attacked. Privacy is super uh important to every organization out there. So they each of these solutions have their own security model that you have to uh take into account and that's what's making it really really hard.
So step number one uh was how do we get all this data out of all these systems and just centralize them uh and have you own it yourself. So instead of you uh having to worry about the data in the different systems, it's sitting in cheap open lakes in an open format. So that was step number one. We actually wrote blogs about this and we called it the lakehouse. That was 5 years ago. And you can see here this was sort of the reaction in the market when we put that out.
Uh they didn't believe in it. You can see you know these are different vendors what they're saying. you know uh choose open wisely was one of the reactions basically saying that it's a misguided application of open and really our priorities and our customer obsession dictate against it and I don't want to pick on a particular vendor it was just the general kind of uh reception of this was that way by pretty much everyone all the hyperscalers and so on um and then you know uh as time went on uh you can see now now everybody loves open everybody's talking about it you see the quotes now from all the hyperscalers, from all the vendors that are out there.
Um, and actually we can see now uh there's even books by the creators of data warehousing talking about the lakehouse uh and how that's the future. So that's super exciting. So now you have the open formats. It's kind of industry has accepted that this is the way to go forward. Though I would urge you in your organization despite a lot of the marketing the majority of these vendors the data that you have even if they they say they like open your organization probably has locked the data 99% of it is still probably in a proprietary format inside of these data warehouses.
So I think we as a community have to push this. Let's really push for the open formats and let's make sure that you own the data. Don't give it to data bricks. Don't give it to other vendors. Own it. store it on open lakes like S3 on Amazon or like Azure data lakeink storage ADLS on Microsoft Azure or like GCS Google cloud storage okay store it there don't store it somewhere else okay because then that leads again back to lock in all right so that was step number one but there was one more step that actually I think we didn't quite understand how important it was and it was that of governance so it's all these uh security models that each of these have and I think we actually created the category four or five years ago, but we never really wrote about it.
And I'm starting to believe that this is more and more important than ever. Uh so we'll be talking a lot about this uh next few days. And tomorrow, make sure you catch mate's keynote. He's going to be talking about this. So this is what we call unified governance. So what is unified governance? Unified governance isn't just let's do security on our data state. It's two very key ideas. The first important idea is that you have to do governance on all of your data assets in the organization.
Okay? And you should do that centrally in one place. Okay? If in your house, you wouldn't have different vendors do the security on different floors or different doors, right? That wouldn't make any sense. It would be very hard to coordinate and make it work. So, you need to do it for all of the different data assets. Um, not just the structured data that you have, but also for all the unstructured data that you have. uh but not just for the data but also for the AI models that you have but not just for the AI models also for all the dashboards that you have.
Dashboards is the easiest way today in an organization to basically excfiltrate data. You get your hands on a dashboard it has data sets you can email them around and then data evaporates out. Uh so you really have to do governance on all of these because the raw data becomes structured data goes into dashboards. We want to track the lineage of all that. Um so that's step number one. That's idea number one. it has to be on all of the different things and really no governance solution today out there uh outside of Unity catalog really does it for all of your data assets.
So that's number one. Number two, uh you want to actually have unified capabilities on top of this data. So what I mean by that is it's not just about security. It's not just about access control. It's equally important uh to have a platform or a governance solution that really helps you with discovery with collaboration because again and again people are recomputing and coming up with the same data sets in the organization again and again and again and then people bicker about why their numbers don't match.
Um you want to track the lineage of all these different things. Uh you want to understand the business semantics. Where do I find the authoritative data set for revenue for this product line? You want to be able to easily find that and there should be no doubt which data set that is. That is what business semantics is and we'll talk a lot about that at this conference on how we taking the next step towards this cost controls.
If you're going to have all this data and AI infrastructure, all these different services, you have to be able to do cost control. So there has to be a way where you can do that centrally. And then data quality monitoring. Uh so these are super important and unfortunately today actually despite all the talk about governance in the market most focus is just on access control on structured data like if you look at efforts like Polaris they're really just focused on this narrow slice uh of governance.
Uh so we're really hoping that you can push your organizations and we can together push towards unified governance and at data bricks uh unified governance um that's really the lakehouse. Okay, we get um at the bottom the open formats and I'm super excited to share now that database 100% supports as formats delta lake and iceberg. You can use both formats. We have 100% support for both. So a year ago we announced that we acquired a company called Tableabler.
Um the founders of that company are the original creators of Apache iceberg and we've been working this whole year on bringing these formats closer and closer and closer. So in data bricks essentially the differences are largely uh now uh negligible and you should listen to mate's talk tomorrow because if you're actually using unity catalog uh as your catalog of record for your open formats whether it's delta or iceberg you will be able to actually read and write to these data sets from any system out there.
Okay. uh and we'll show in performance benchmarks that if you pick unity catalog as the catalog of record for your open data in Delta Lake and in iceberg you'll get significantly better performance in actually ingesting and processing that data. So this will be the most open approach you can take uh to the lakehouse and then on top of that we provide Unity catalog which we open sourced last year. We're continuing that.
We're going to continue open sourcing uh more and more. Uh and most importantly, it also implements two of the standard open source interfaces for cataloging. The first one being Hive Metas store and the second one being ice rest catalog. So these two interfaces, any system that wants to talk to an open catalog, you can just pretend this is a hive metas store or this is a ice rest catalog. And it's actually the most complete implementation of ice rest catalog in the industry today.
Um, so this is the lakehouse and we're super excited about this. This is what we've been focused on for the last five years. But what we're really excited about at data bricks is when you take this lakehouse as your foundation and then on top of that you build intelligence into the platform. Uh so what we mean by that is you take AI and you infuse it throughout the platform. That's what we call data intelligence. Okay, data intelligence for us means really two things.
One is we want to democratize access to data in your organization. In the past, you had to be someone who knows Python or SQL or maybe even Scala or Java to get value out of your data. We want to challenge that and change that so that it should be enough if you just speak English to your data or any other mother tongue or natural language that you have and we should be able to get answers for you. So that's number one for data intelligence.
Number two is we want to also democratize AI which means we want to help you build your own AI for your companies so you can innovate AI that can reason and answer questions on your proprietary enterprise data or organizational data uh that's out there. Okay. So not just general intelligence and artificial general intelligence that can answer questions about anything out there but that really with high quality and uh adequate costs can answer questions on your proprietary data.
So on the left hand side democratizing data uh the sort of you know English the new programming language we have two products that we have had in the market now for over a year. Uh Genie on the left hand side you can see um is a product that's now used by 81% of our customers. Um how that works is that you just ask your question you create these data rooms. So you can take any data sets in Unity catalog and create a data room and can share it with anyone in the organization invite them to the data room and they can just speak English to the um to Genie and what it does is it uses uh an ensemble of agents comes up with a plan uh and actually writes code for you and then executes that.
So you don't need to know any of that you're just speaking English and it's just interacting with you and giving you the results that you see on the left hand side. So I think this is going to be the future eventually. this is going to be the way everyone does data science but today it does require preparation of the room if you don't prepare the room uh you know generative AI still has ways to go for it to be completely automated right so um you know there's you know still can't do the full thing that data scientists do unless you've prepared the room on the right hand side is the assistant so 98% of our customers are actually using the assistant and what that is is you know throughout data bricks we've made the assistant understand every aspect of the platform so you're running into an error, you can click diagnose error.
If you have a piece of code, it can explain it and it understands your data in the organization. So that's the key thing about the assistant. Uh unlike other assistants out there that you know are great at programming, this one understands the data that you have in data bricks. So that's what we call data intelligence. Um I've been a power user of data bricks, no surprise, for the last 1015 years. And today I just use this all the time.
Like this is my preferred way. Even though I used to prefer just coding myself, I use the assistant. I let it write the code for me. Uh I review it. Uh but it's completely transformed what I'm doing and I'm much much more productive and I think we're seeing it with everyone that's using um data bricks. Okay, so that's just democratizing data. Let's look at AI. On the AI side, we announced last year uh Mosaic AI agent framework.
You could build your own agents. You could do vector search. Uh you could do model fine-tuning, guardrails, and so on. And in particular um we have now if you look at the platform classic ML is being used by 95% of our customers and then we have generative AI that's picked up and it's at 81% of our customer base. Uh so super excited about this. Um listen to Hanland's talk today where we're going to talk about basically the next uh generation uh of how we want you to develop your own data intelligence on your data.
So one of the big announcements are going to be by Hanlin. Okay. Um, so that's the data intelligence platform. That's what you're seeing in this picture. You have the lakehouse and we've infused AI throughout it. We've simplified everything by removing a lot of the knobs using AI. Uh, and we think that this is going to be a big deal. Data intelligence uh is going to uh be something that every organization on the planet wants.
Uh, and that's going to change and transform jobs throughout the planet. And I think we need to educate uh billions of billions of people around the world on how to do data and AI themselves. That's why I'm super excited to announce today that we're actually launching what's called the free edition of data bricks. So, so what is the free edition of data bricks? Free edition of data bricks means you don't need to swipe any credit card.

`20:30` Introducing Databricks Free Edition – Learn and explore with data and AI for free on the same platform used by millions of professionals

We're not going to ask you for any credit cards. We promise. Uh and we're not going to ask you for a business email. uh you can use your Gmail or your Hotmail or uh you know and just log in and you can get a free slice of data bricks forever. Okay, it's just there. Um but the limitation is you get a small slice, right? You can't come in and say I want a million machines because then we go bankrupt in an hour. Uh right, but that's how it works.
Uh and together with this we're also doing a hund00 million investment in training and education and try to get databicks free edition out to all the universities out there. So please help us and we've taken all of our self-paced learning that we've had over the years where you can learn to do various things like you want to build a recommener or you want to do inventory analysis or you want to do forecasting or you just want to build data pipelines that are robust.
We're taking all that self-paced training and we're open sourcing it and making that free as well. So really need your help. Please help us get the word out with this. if you have contacts at universities or if you want people to learn uh let's get this out there. This is the best way for anyone uh to learn and get started with data and AI. Okay. Um so super excited about this. Data democratization is super important.
Data intelligence is super important. Uh so uh I want to welcome on stage uh Richard Masters who's VP of data and AI at Virgin Atlantic to tell us about how he's democratizing data and AI at Virgin Atlantic. know, it's a long walk to the stage here. , hi. Super excited to see you again. Great to be here. Um, so I'm a little bit curious. So data is the foundation of all this stuff. Uh, I'd love to hear a little bit at Virgin Atlantic.
Uh, how do you get the right data foundation so that you can do all these exciting things on top of your data? Yeah, it really starts with my great team really. We got data product managers, data engineers, scientists that really build data assets and products on the Medallion architecture through bronze, silver, and gold and all managed by Unity for us. That makes it all visible for us. We can see it all there on the on the on the left side of the screen.
You got the flight, you got customer, you got commercial. Um, and it all becomes that left join of the organization. But it makes it discoverable. It makes it sharable. So we can use Genie, we can use assistant, we can use dashboards to really help our stakeholders out there in the different domains use the data, discover it and trust it. Awesome. So step number one, get the data in a good place and you need a great team for that which you have.
Uh, that's awesome. So then once you have that in place, uh, how do you deal with security governance? How important is that first of all and how do you do that on top of all of your data? Yeah, it's it's super important. The way we think about governance is it's it's intrinsic to each product that we build and whenever we have a product or or a problem or an opportunity, we start with the requirement. We build that out and that's got requirements around the insight that has to come out of it or or the integration that has to happen with enterprise applications.
But through that is the security, the type of data underneath it, the observability of costs, management of that. It all helps us through Unity catalog track that right through to the compute. uh right up to the to the output the way it's going to be consumed uh and we can we can offiscate we can um mask out we can really track that and give trust in that data in that output through the governance of unity catalog. Awesome.
Okay. So we have the medallion architecture you have the data your data teams fix it and then on top of it you have your security you've done your masking in unity catalog so that the data is ready uh to be consumed. Uh are there any exciting use cases? Are you doing anything with data intelligence AI? Uh tell us a little bit about the use cases. There's there's so many use cases and exciting problems in the Aon industry.
We've got a lot of traditional ML problems around predictive maintenance, around dynamic pricing, understanding drivers of experience for our customers. All great stuff. But some really exciting stuff we're doing more recently with language models is around triage is around summarization and categorization of health and safety messages. You know, super important for us as an as an airline to keep everyone safe. And we want to be able to we want people to report but we want the teams that are dealing with those messages to really focus on fixing things or anticipating things that might come up.
So what we're managing to do with language models is categorize, summarize and prioritize those with the human in the loop. So this is where our visions really align at Virgin Atlantic and Data Bricks where that human in the loop is is critical to how we really engage with AI in the organization and it's saving those teams, you know, one to two hours per day, several hours per week. And we also have an engine that we can reuse around multiple other customer care use cases as well.
So that triage component can really be reused around everything in the organization. That's super impressive. So basically we can make the airplane safer using AI because the whole triage process can be automated with a human in the loop. Whether that's a big thing for us at data 2, we'll be talking about it next few days, which is we think that for the AI there needs to be a human in the loop because they do make errors and we need to make sure that we're supervising that and making sure that's correct.
So, uh, super excited about this. Thanks for sharing this with us. Thanks. Thanks so much. Round of applause for. Thank you. Okay. So, um, we know that AI is going to disrupt things. I mean, you heard, uh, triage management, lots of lots of different use cases in every organization. You're seeing it. I'm sure it's the number one thing that's being discussed at your company. Uh, and you can see some of the headlines here from the newspapers.
I mean, there's like every day it's like endless number of these. Future of payroll, content management, uh, CRM, IM, and so on and so forth. In fact, if you look at all the different categories of software that exists today, whether it's, you know, CRM, ERP, human capital management, contracting, and so on and so forth, uh it doesn't require that much imagination uh to see how AI is going to completely disrupt each one of these.
AI is going to be infused into this. You're not going to have to manually go and enter things into the system. It's going to be able to capture all of that. It's going to be able to organize a lot of that and it'll be able to give you the insights automatically. Just like the triage example, you'll know which customers you need to upsell. Um, you'll know exactly what you need to do in the supply chain. You know, where the bottlenecks are, where to place them, you know, which employees you're going to promote, who needs uh a raise and so on.
All the software is going to be disrupted by AI. So, the question is what's the foundation needed for each of these uh categories to be able to do that disruption uh with data. Okay. So really at this conference we've been talking a lot about data and AI right we said that's clearly two very important uh pillars of all of these applications the future disruption will build on these you know every one of those categories every one of those companies companies that maybe you will start or companies that you are in will build on that data and AI foundation and at data when we usually say data what we really meant was basically data analytics okay that's what we kind of meant right data warehousing and you know the management of that data uh for anal analytics purposes and then we had the AI.

`27:30` Announcing Lakebase Public Preview – Fully-managed Postgres for data apps and AI agents

Uh but one thing that we haven't talked too much about at this conference in previous years which underpins actually every piece of software that exists around the world uh is the database. Okay. Okay. So I'm here I'm talking about the traditional transactional you know database you know the SQL server Oracle MySQL Postgress kind of uh underpinning database and we think that's going to actually change significantly and it's a really important area going forward uh with the infusion of AI and what's going to happen next.
Okay. Uh so uh if you look at this these databases they were really built for a different era. um you know te technology in transactional databases hasn't actually changed that much in the last 40 years. Uh in my previous life uh at we used to look at this paper called system R that was published in the 70s. There's a research paper by IBM and 90% of all the ideas that you find in the database today are already in that research paper from then and not much has happened since then.
Uh so why is that? Why why is why are these databases uh looking the same as back then? Well, number one is that because data is so sticky and once you've picked a transactional database and you put your data in it, it's just nearly impossible for you to move off of it or move it to a new thing. So there's just this massive lock in on all traditional transactional databases. And once you've locked yourself into that, those vendors also don't need to really uh innovate.
They don't need to challenge status quo because you can't go anywhere with it. Okay? So it's just such a sticky thing much more so than anything else much more than data warehousing much more than the kind of stuff that datarix has been doing uh for you know the past decade. Number two, they're very expensive because of that lock in. Once they have your data there, you know, they can raise the price. And also the architecture, the way it was built uh in the 80s and 90s.
These are, you know, big black boxes kind of maybe they're virtualized in the cloud, but you still clunky big instances that you buy takes a long time to spin them up and, you know, they're not really elastic in their foundation. Uh and finally, they're really built for that on-prem era, right? They don't have cloud, they don't have AI in mind. They were built pre-AI. Okay, so what we're really excited about at data bricks and this is another one of our big announcements is that we think that there's going to be almost a new architecture for these databases going forward like almost like a new category.
Uh we call this the lake base and you know might sound familiar. It's you split the database into a base and a lake layer and you take the data that's sitting in these traditional transactional databases and you store it in lakes. Okay, in cheap data lakes uh in an open format. Okay, so basically disrupt that transactional database. You should own that data. It should be on the lake uh in your account and then on top of it you do the transaction processing in the base layer on top.
Okay, so that's what we're super excited about. A simple idea but kind of hard to pull off and we think this is going to be the future of all databases. So let me tell you a little bit about the lake base. Um so number one we think it should be built on open source right uh and we think that postgress essentially has one as that lingua frana for uh open source databases if you look at its traction it's absolutely massive and the best thing about postgress is the extensions that the community have built over the years.
So it's like a massive ecosystem of extensions that are available that lets you do anything from you know if you want to do vector processing or key value store or whatever you want want to do it's part of Postgress open source extensions that's number one number two uh you should really properly separate compute and storage in that database layer in Postgress and this is really hard to do because unlike data warehousing or the lakehouse um you want this to be really low latency okay so just doing every query and going to the lake would be too slow if you do it natively.
So, it's been previous attempts at doing this, but you know, they don't really properly separate it out. You want high QPS queries per second, millions, uh, and you want to production SLAs's because this is going to be underpinning your most important precious AI applications in the future. Okay. And then finally, um, it should really be built for the AI era. Okay. It should be able to launch very quickly because we're going to see a lot of agents launching these. uh and we want to be able to make sure that if they launch hundreds or thousands of these databases, you should only pay for what you're using.
Okay? Like this should be able to scale to zero and most of the time you're just paying for cheap data lakeink storage. Uh so that's what we think is the lakebase. We are super excited about this. To talk about more about this and what we've been doing at data bricks for the last two years to work on the lakebase, uh I want to welcome my co-founder uh to the stage. Um, thank you. We're actually supposed to shake hands, but given you walk all the way there and I'm supposed to come here, I think it'll be a little bit difficult.
So, virtual handshake. Um, the as said, so one um data bricks in the last decade has mostly been focusing on the analytics side of data infrastructure. Um and if you look at analytic systems today, what you use on data bricks or even on on other vendors, they look remarkably different from what they were in the 90s. Um there's a lot of foundational technologies been invented. Commized processing have dramatically speed up analytical workloads.
Streaming was invented about maybe over a decade ago actually make data a lot fresher. And of course in 2020 five years ago we published uh the lakehouse blog and pioneer new architecture which decoupled storage and compute and more importantly based on open formats and that have enabled a lot of new workload dramatically lower the TCO for all the analytical systems. The now OLTP databases however are kind of stuck in the past.
Um what do we mean by that? If you look at OLTP databases running today, whether it's commercial super proprietary systems like Oracle or open source databases like MySQL, Postgress, they look more or less the same as they were in the '90s. Yes, we've added a lot of features. They have gotten faster. We've looked at the components and the techniques and foundational ideas. They're more or less the same. Um, I even sort of referred it all the way to the 70s with the systemr papers.
Um and databases are viewed as this heavyweight infrastructure requires a lot of manual intervention and maintenance and it's quite clunky. Um first databases are very slow. So the provision and it's sort of difficult to scale. Um if your workloads are fairly dynamic it's often a nightmare to actually deal with that in the database side. Um and because of that um databases are fairly disconnected from modern day developer workflows u which we'll zoom into a little bit and it's also very silo from analytics and AI like it's actually not unusual these days you want to combine analytics and AI with your transactional database workloads uh but it's very difficult to do so what did I mean by uh databases that disconnected from modern day developer workflows well imagine you're a software engineer and you're trying to add a new feature to a codebase the very First thing you'll likely do is the following command.
Get checkout-b maybe click in the UI. Um, but what it does, it creates a new branch of your codebase and you make changes to this new code branch. Um, adding new feature, maybe fixing a bug, you'll be testing against it. But all the changes you do are isolated to this specific branch. And creating a new branch is an instant operation. It's very, very fast. You don't have to think twice about it. You just do it right.
What's the equivalent for databases? You want to clone your production databases. It might take days. Um, you put one up, you almost never shut it down. Some things like this simply doesn't exist. Like, wouldn't it be nice to can branch off a database just like you would do with code? Now, let's say you get past all that development hassle and you manage to build a pretty successful app, which many of you have. Um, and now um the app had taken off.
You want to add introduce some analytics capabilities or AI capabilities to the app. So you have your app development team now start talking to your data infrastructure teams. They're managing the lakehouse. You say, "Hey, how do I actually get the data from one side to another?" Um, now you have to figure out how do you manage two disparate systems? You have to understand the IM role differences. How do you set up secure networking?
How do you create ETL pipelines and load data from one to another? You learn fancy terms like change data capture, std type one, std type two, which I up until this stage still don't understand what they are. um it just seemed awfully complicated. The so in the past couple of years at data bricks we've been working on how to tackle this problem and actually eliminate all the challenges and the result of that is lakebase right lickbase um has the following attributes first and foremost it's based on open source progress um and second it built on a novel decoupled storage from compute architecture that actually enables the modern day developer workflow um and by building on top data bricks infrastructure it comes with what you would expect actually amazing lakehouse integration as well as all the enterprise readiness features.
Now let's talk about them 101. So one each after each. So first and foremost, Lakebase is building on open-source standards which is open source Postgress and in the last few years um open source Postgress have steady on the rise and if you look at the latest stack overflow survey of the most popular databases Postgress actually leads by a wide margin um and this is because it's robust ecosystem of tools libraries and extensions and all this just work out of the box on the lakebase and lakebase can guarantee you single-digit millisecond latency at scale.
The second most important part to Lakebase, it's building on a new novel separation of storage from compute architecture. And it actually has three layers to this architecture. At the very bottom, we're using data links or object stores to store the actual physical data. And object stores are the cheapest storage medium you can find. They're extremely reliable at scale. Now, one of the challenge that actually referred to is that object stores were not exactly designed for the type of workloads that OLTB databases need.
A 100 millisecond query is plenty fast in a lakehouse for analytics, but 100 milliseconds is actually unacceptable for OLTP workloads. We need singledigit millisecond latency. Um, so the way we've solved that is by introducing a middle layer storage that's actually only have soft state and it acts as a right through cache for all the data to the object stores. And for some of you that are database nerds, it also um creates a a new way to very quickly persist the right headlock or we typically call as wall um in the database.
And on top of the storage layer, we have the ephemeral compute nodes which are Postgress instances um that actually reads and writes to the underlying storage layer. Um one thing that's very important for me to point out is very similar to lakehouses. The actual data that store in the object stores at data lakes are open source formats. They are not some proprietary format we invented to actually improve performance or to lock you in. they are just vanilla Postgress pages and this opens up a whole new sort of paradigm of opportunities.
Um and there's actually a lot of past attempts at this problem. This is not the first time I think um sort of the industry have tried to create a separation of storage from computer architecture for OTB databases. Some commercial systems especially from hybrid scalers have done that. Um but they were typically actually built on yet another proprietary storage system. They are way more expensive and also don't use open source formats which means they can manage to lock you in even more.
So um the way we crack the code here um is actually based on um some of you might recognize sort of the architectural diagram and data bricks actually acquire um this company called Neon just last month um and you're already talking to us about it. Um now we did acquire Neon. We announced acquisition last month. We actually only closed the acquisition last week. uh but one of the interesting things that's not widely known is we actually invested in Neon the company uh many many years ago and been working with Neon team as a technology partner on this separation of storage from computer architecture and everything on lakebased actually building on this collaboration right and building on top of this new novel architecture we managed to accomplish few fairly interesting things the first is civil and what do we mean by civil well earlier we said database are this heavyweight infrastructure that requires a lot of manual maintenance and intervention.
Um serless here means database should become lightweight. Um what does it mean it's lightweight? Well, lake base come in two flavors. The first flavor is a provision throughput flavor um flavor that it's it shows you you actually specify exactly how big you want it to be. If you know how to size your world, that's the perfect solution for you. But for most people, the autoscaling flavor will be far more interesting. In the autoscaling flavor, you don't have to worry about um how big a database you should be picking.
And because the databases are actually just ephemeral instances, you could only launch it when you need to. It takes less than a second to launch a brand new database. Um and if you're sort of low scale, you could either vertically scale it, which the system will do it for you, or could choose to create re replicas that also comes up in less than a second. Um, and if your lows goes down and you actually no longer have need, for example, say for a very American centric company u past 5:00 pm, maybe you have no loads, you can actually automatically shut it down very quickly.
All of those just happens in less than a second, right? And the best part is to only pay for when the duration you actually need the compute. Um, the second thing we built was branching. Uh we talked about earlier how difficult it is to branch off a database and to apply modern development software practices to actual databases. It's very easy to do it with code but it's very difficult to do with databases. Um the separation of storage from compute architecture also has a copy on write um capability built in that we can instantly branch off a database.
It takes less than a second to create a whole clone of the database for and that includes both the data and the schema of the database. Um, and because of the copy on write capability, you don't actually have to pay for extra storage unless you start making changes and only the changes themselves will incur extra charge because under the hood, they all share the same storage. Um, so something pretty magical would actually happen when you combine the branching capabilities and the serless capabilities into one.
It would completely change the way you think about database development. Every time you do get git checkout-b you should automatically actually branch off a database with that new branch of code. You should have them perfectly sync in sync making schema changes. If you actually don't like your new code branch and whatever changes they make to the databases just kill both the code branch and the database. You pay next to nothing just like how you're paying for your code repository.
Right? And this is extra important in the age of agentic coding and AI. um with AI like one way to think about AI agents is you're um getting at very low cost an armies of thousands or maybe even in the extreme case millions of AI agents and the AI agents are acting as own individual engineers that doing experiments on your codebase maybe adding new features you might even have multiple AI agents adding new features um adding the same feature and you have judges to determine which feature is the best implemented and it's now every AI agent can actually get their own co- branch but also their own databases at virtually no cost for experimentation.
The separation of storage and compute um especially by leveraging actually open source formats underlying storage layer also makes it super easy to synchronize data at very high throughput from one object store to another object store. So from one data lake to another data lake from lakehouse to lake base. Um and many of you if you're existing data bricks customer given what we're doing you probably actually expect this out of the box.
Um you can publish any tables in the lakehouse into lakebase for real-time serving to get your template second latency and you can also do the reverse. can very easily get the data from lakebase directly into lake house managed by UC MU community catalog with a few uh clicks and of course by building on top of data bricks infrastructure um lake bases are uh sort of enterprise ready it comes with all the wells and bles you expect um from security to compliance to governance so given lakebase what can you do differently well first if you're trying to build a new app and you need a relational database.
Give leg base a try. If you want to serve data, you have data today whether it's machine learning feature stores u or whether it's a simple um data pipelines you build you want to serve that data um give lickbase a try and if you have ETL pipelines complex ETL pipelines that ingest data from a relational database into data bricks which I know almost every customer does um give lake base a try it'll dramatically simplify your architecture so with that I would like to invite onto stage to actually give you a demo to uh visualize what looks about and I think Holl's actually going to come up on that side.
So you have slowly shifting the stage. Hello. Uh, I've been given the job to manage inventory levels for a drinks company, making any last minute adjustments to stock, but also to share data between analytics teams. This job is tricky at crudtime. But fortunately, I have some new tools from databicks to help me. And today I'll be sharing how Lakebase powers databicks apps, works at scale, and can use data from Delta tables all in real time all in one platform.

`46:40` Lakebase Demo: Holly Smith – Lakebase

So I'm going to switch to my demo and in front of me I have a data bricks app that's bringing together both operational and analytical data. Whenever I'm changing these filters these are live queries. Nothing is cached in the app. I can see here orders that I need to review but I am not going to do that. Uh instead I am getting some intense requests to place some orders for cherry burst for 90 units. I'm going to select my store click through the app and go.
This app will go and update Postgress in the back end where the orders table sits. Now I'm going to double check that this has gone through correctly and I'm switching to the brand new Postgress SQL editor natively integrated to the data bricks platform and I can tell it's Postgress because I've got this icon on the top right here. Um, and when I select my compute, it's got this Postgress tag next to it. I'm also not going to do this on production.
Uh, so I am going to use dev, which is an isolated test branch. And if I look at the left, I can see some neon details. Um, I'm also going to check the settings of the version. And I can see that yes, this is Postgress version 16. 6. And to query the data, I would just query it like I would any other table. And I can see here that when I run this that yes, my order has gone through. But that's just one record and that's not very impressive.
So earlier today, I set up a little program to simulate lots of users to run in the background. Um, I'm going to switch to the native performance monitoring in data bricks. I'm going to head to this metrics tab here. Um, and I can see how this has ramped up to thousands of transactions a second. Uh, and tens of thousands of rows per second. And I've got all sorts of interesting operation types happening here. One thing this doesn't have is our app latency as it scales.
And fortunately, uh, I bought my little program so it can measure latency. So, I'm just going to run a simulation and see how well we're doing. And oh, I could see my performance is almost 19,000 queries a second. Uh, with the median at 4. 56 milliseconds and the 95th percentile at 5. 6 milliseconds. So, I'm pretty pleased with that. Thank you. Lakebase can also access data in Delta and I can combine them together for snappier responses in the UI.
So, I head to the table in Unity catalog. I can create a synced table. I go ahead and I can name the table. I select which instance I want to use. I want to use production. I can also choose the database as well. And then the primary key. Now I can either run this as a one-off snapshot triggered at a certain interval or have it continuously updating. And I can either group this with other updating items in a pipeline or I can have it as a standalone.
Now that's set up, uh, when I go to it in Unity Catalog from the database view, uh, I can see here that I've got this little synced icon next to my table. And so now when I head back to my app, oh, here it is. And I head to my insights tab, I've got this insights tab, and that is including data from a delta table for my forecast. And of course, I can sync all of my orders uh data back to the lakehouse for historical analysis and the other way around, too.
So, in this demo, we've shown how data bricks is bringing together both operational and analytical data closer together. And I've hope I've shown you how powerful this is when combined with other data bricks features like apps, delta, and unity catalog. And it's inspired you to give it a go on your next project. And as for me, I don't think I like being an inventory manager and I probably should have made this an agent.
Back to you. Thank you. And only virtual handshake. Um, so at data and AI summit and other conferences, we're announced products of various stages of maturity. Where where is lickbase? We just spent so much time talking to you about it. So in the last year we actually been private previewing lakebase with hundreds of customers and many of some of their logos actually showing on the stage here uh across a very variety of uh industries many of them actually running lakebas in production and we're also very happy to have the following launch partners joining us to uh announce lakebase itself and this includes catalog vendors BI vendors agentic coding platforms and consulting services but the best part is that lakebase is available today.
All right. So, not something coming next here for today. Starting today in all of your data bricks workspaces, depending on which region you are, you can either explicit opt in or it's already on out of a box for you. It includes the full-blown fully managed Postgress instance, all the lakehouse integrations, multi cloud support, HADR. All right. And there's a lot more new features coming in the coming months. Um, so just to summarize, Legbase is offers a fully managed Postgress instance.
It comes with a new new novel separation of storage from computer architecture which enables the modern-day developer experience both for humans and for AI agents. And more importantly than the lakebased product announcement, we actually feel like this is how database should be built in the future. And our prediction is every other database, every other transaction OLTB database will evolve towards this architecture in the coming years.
Um, now we can't not talk about Lakebase without talking about Neon acquisition because that's what power the innovations behind Lakebase. So with that, I will actually like to invite the CEO of Neon, onto stage um to talk to us. Nice shirt. Thank you. Thank you. Um so um you started Neon together with stats and hecki about four years ago and at the time and neon became sort of the leading serless Postgress company but the time there's a lot of other Postgress companies and all the hyperscaler had postgress what problem were we trying to solve?
So we actually focused on uh developer experience of all the things. The biggest neon innovation is uh separation of storage and compute which does allow us to store data in an open format inside of the cheapest medium uh cloud object stores and you know lakehouse. Now what does it have to do with developer experience? Well actually a lot. So the one thing that it does allow you to do is to make a serverless system and without separation of storage and compute you it's actually kind of impossible to make a system serverless especially a system that is kind of a traditional database like postgress.
So now you can think about a database as just being a URL right and this URL is something you application interfaces with and then according to the application it will increase the size or decrease the size or scale all the way to zero if you stop uh driving that system which is very convenient. You don't have to think about sizing and you only pay for what you use. The other thing that we we quickly realized when we kind of put a magnifying glass over the developer experience is that developers like to integrate their systems in their developer workflow.
And that's where the idea of branching came in. And again, it's only possible to deliver on branching when you branch both schema and data if you separate storage and compute. So now um to do it instantly at least. Yeah. And to do it instantly as well, right? We use copy and write for this. Um, now copy and write is not a new idea, but packaging in in in a way that developers can self-s serve consume was a big innovation from Neon.
Makes sense. Um, one other thing that was actually very surprising to me at least is um we noticed um throughout the process, hey, 80% of the databases created on neon. com um were created by AI agents, not humans. So AI agents actually create four times more databases. And here's sort of a chart and it started at 30%. what's going on there? Can you tell us more about that? Yeah, that's right. Um, so so last year um we had 30%, now we're 80%.
I actually want to make um a kind of bold prediction. In a couple of years, I think 99% of all the databases on the platform will be created um by AI by AI agents. I I think we're at the dawn of the AI software revolution and every engineer is becoming an AI engineer. What what this means is if it's a human uh then the human is levered by this modern tools such as cursor a windsurf or Microsoft copilot and um but also we now starting to have uh fully autonomous AI software agents agents that generate software um you know specifically um partners of data bricks uh replets and lovable that are that are here part of the our launch partners here um they they just generate apps from a prompt and since every application requires uh needs a database um neon is a perfect solution for that because again database is just a URL database has branching and um AI kind of makes mistakes.
So creating those isolated environments uh in a safe and secure way is a big deal for those AI systems which are really hard to do with traditional databases impossible. Um, last question. Um, what are you most excited about now that Neon's part of data works? How do you see the text coming together even though they kind of were coming together already? Right. Um, well, I'm excited to build the best OOLTP system in the world that is fully integrated uh with the uh data and AI platform.
So I I think the um analytical and OOLTP systems um they have been separate uh for a very very long time and there are all these use cases that are pure analytical pure transactionals but then there's also anything everything in between. So I'm excited to not not only build the most um you know scalable and efficient OOLTP system in the world but also deliver all the use cases for all of you that span the whole range between analytics and OLTP all based on open source all based on the source.
All right. Thank you. And Ali, we're going to pass back to you. Awesome. Super excited about this. Uh and very very excited to have uh and the team uh as part of data bricks. Over hundred of uh you know Postgress folks that are joining us. So we really want to embrace the Postgress community as part of this conference. Uh and we're looking forward to the innovations that are going to come uh on lake bases both at data bricks but also in the community at large.
Um okay so um one thing that I'm really excited about by the way I forgot my clicker so in the back you guys have to click uh for me. Okay when I say click that's what that means. Uh um so all right so we got you know remember the three uh kind of pillars of all of future AI applications. um you know we have the data analytics uh we have the AI uh and then we have the database so we're kind of on the database leg here uh and so at data bricks we discussed how are we going to help people democratize access to the data they have in data bricks is there a way in which we can help them authenticate and get the access controls and as easily as possible get access to that data directly so uh you know a couple years ago there was the discussion that we wanted to really democratize things in data bricks with what we called data bricks apps.

`58:30` Announcing General Availability of Databricks Apps – Build Secure and Governed Data Intelligence Apps

Okay. Uh and we had a big discussion internally and there's always debates at data bricks you know how should we prioritize things and I was kind of a skeptic I said it takes a long time writing these apps is going to take a very very long time you know it's you know how much resources are we going to put on this uh and we actually made a bet internally and I said that look I didn't I think that in the first year you know maybe we'll have some hundred of organizations using this click uh and I was totally wrong.
So we launched this in November 2024. So data bricks apps and we have over 2,500 uh customers actually building their own applications you know v coding them building them and the beauty of it is that it just integrates directly with data bricks. So to find out a little bit more of you know why I was wrong and what databicks apps is about. I'm super excited to welcome on stage the braant uh from databicks to tell us about databicks apps.
Let's give him a round of applause. Thank you. Wow, it is good to be here. Um, and let's uh let's see if we can convince all of you why apps is the way to democratize data and II in your organizations. You can see that a believer. Um, and he was a bit of a skeptic. He's not lying. So, let's get into it. So as said um we have the foundation for modern data intelligence applications. We've always had the analytics that's been the foundation of data bricks from day one.
Um we layered that with AI and now through lakebase we have the database as well. We have the foundation and now it's about what goes on top. So great everybody's there out there has probably done some sort of vibe coding. It's easy to build a front-end application now. I can just vibe code up my app and deploy it. Well, the hardest part is actually connecting these apps to the data and AI in a secure and governed way.
So, it's the productionalizing of these applications next to your data and AI that is really challenging. And anybody who's done this knows there's some aspect of all of these that you have to do. You have to integrate with your off and SSO. You have to define governance and access controls, manage the infrastructure, containers, integrate into your VPC. So all the networking stack you need to deal with, deploy a database to manage state or connect operational data, integrate with observability tools, configure audit and access logs, everything that goes into productionizing is actually the really really difficult part and that's why we are super excited to announce the general availability of data bricks apps.
Thank you. Thank you. Databicks apps makes it easy to build secure and governed data intelligence applications deployed directly in data bricks and governed through Unity catalog. And really this is about a broader architectural shift of moving the app to the data and AI versus moving the data and AI to the app. This is really really critically important and we think this is the architecture of the future. So let's get into it.
Why data bricks apps? One, it's powered by the databicks data intelligence platform. Everything in data bricks becomes a building block that can power these data intelligence applications. You have out of the box off and SSO. So you can bring your own data bricks identity to the app and not need to reauthenticate and reintegrate. And as of today, we're announcing a one-click integration with Lakebase. So you can easily deploy your data intelligence app with a database backend.
You can use that for app state. You can use that for agent state. You can use that for operational data. It's easy to deploy and integrate. Secure and governed. Apps is hosted in data bricks managed containers and infrastructure. You don't need to spin up and manage your own stack. Everything is governed through Unity catalog. So you can bring your own permissions to the application and access your data as you would as an individual user, even down to the individual resource level.
And finally, we have out of the box off and access logs for enterprise auditability, open ecosystem. This is something that data bricks talks about a lot. We believe in openness. We believe in open source. We believe in open frameworks. We took that same philosophy with data bricks apps. We supported out of the gate all the popular Python frameworks. Streamlit, gradio, dash, shiny, fast API, flask, whatever you wanted to build in, we support that.
And as of today, we now are announcing support for JavaScript as well through node and react. So it's not Yes. A lot of JavaScript developers. I love it. So now you can choose your own framework, you can choose your own language, you can build the app wherever you want to build it and then deploy it as a data bricks app. And in addition, we support 20 plus pre-installed open source packages. So you have everything you need to build the best data intelligence app already deployed.
And as said, uh the adoption and the customer feedback has been incredible. Um, since our launch in November, we've had over 20,000 apps created across over 2500 customers. And that actually makes Data Bricks Apps the fastest growing public preview product ever at Data Bricks. Thank you. And I know a lot of the customers that have been part of this journey with us are in this room. So, thank you. We're just getting started.
And let's take a look at a couple of them now. Bridgestone Bridgestone uses data bricks app to operationalize foundation models across their organization. They do prompt caching, persisting history, token consumption. They operationalize the use of LLMs within the organization using a data bricks app. And this app is actually used by hundreds of internal users weekly at Bridgestone. So it's not just a simple personal app.
This is organizationwide. These apps are being leveraged. Adapar Adapar has streamlined ingestion through a custom interface to empower non-technical users, non-data science, non-data engineers to configure data pipelines and run ingestion. This is actually a really important pattern that we're seeing because it's allowing data science and data engineering teams to build instead of support. I'm sure many of you in this room spend a lot of your time day-to-day in ops one-off things.
Run this query, run some job, run some model. It's cumbersome. It's burdensome. Apps allows you to create custom interfaces to make that self-service. So you can get back to building and innovating and not just being a support team. Teada, the global pharmaceutical company, uses data bricks apps to process investigation summaries. They take this unstructured data. They extract key information from that, put it in a structured format that's more easily consumable across the organization and generate summaries.
Again, they're operationalizing traditional ML and foundation models through a data bricks app. So, that's super exciting. Um, actually, you know, my favorite part about my job is I get to get on calls with all these customers and I just get to see a bunch of cool demos of apps. So, this is just really scratching the surface on what our customers are building. Um, and it's been truly inspirational for me to see everything that's going going on.
But what about app development, right? We've talked about apps makes it easy to uh deploy and manage these production apps but what about the actual development of the app and development is changing as suggested um there's a lot going on in the market whether it's you know anyphere and cursor raising a ton of money $10 billion company after only a couple years um you know open ais talks to acquire windsurf uh replet is blowing up in terms of usage and evaluation.
Uh, Retool just announced their own AI app building um, product. There's a tremendous amount, and this is just in the last couple weeks. Um, I didn't have to go back to last year's data summit. Um, so there's a tremendous amount of momentum in the market around development of apps and how that's changing with AI and vibe. Coding is great. I do it in my spare time. It's really fun. But developing production data intelligence apps requires more.
It needs to be aware of your data, aware of your models, aware of the products and services that ultimately you're going to integrate into. Really, you need to integrate this deeper into your data intelligence stack for it to be valuable. And that's why we're excited to announce our app builder ecosystem. The app builder ecosystem is the modern development stack for data intelligence applications through data bricks and our partners.
The idea here is that we're supporting any interface and any framework whether it be AI based coding, visual editing or going directly to the code. You can develop alongside your data and AI with awareness of the products and services that you're ultimately integrating into and then you can seamlessly deploy that as a data bricks app. We're super excited about these partnerships and we're just getting started. And with that, I start the long walk here.
Uh so for this demo, we're going to try to bring it all together. Um we're going to show the development of uh of an app with one of our app development partners, uh Superblocks. Um so we're going to generate that app with AI. We're going to infuse that app with data intelligence from the data bricks platform. And then we're going to deploy that app seamlessly as a data bricks app. And once I catch my breath over here, uh we can uh start the demo.

`1:09:15` Demo: Justin Debrabant – Databricks Apps

All right. So what we're going to do is we're going to copy this prompt in to generate a supply chain dashboard for BrickHouse brands. We're telling it which data bricks products and services to integrate into and it's going to generate the framework of our application um the interface as well as the APIs into data bricks and here we are. So what we can see is now we have our brick house brands admin dashboard. We're going to go from top to bottom and inspect each of these components to see the underlying data brick service it's integrated with.
Starting at the top here, we have um the aggregate timebased information for SLA violations, active SKUs. We can see that this is an aggregate query from DBSQL. Um going out here, we can see the same data visualized as an embedded AIB dashboard. So we can see the timebased visualization and down here we can see the individual skew level logistics. So let's actually click into one of these and we can see now the shipment history for this skew.
This is real-time data being pulled from Lakebase. And up top we can see the priority scores, predicted delays, delivery risk. These are actually queries to a model served through model serving prepared through MLflow. And so what we're doing here, you can see the lakebased query and the model serving query. We're able to integrate the real time data with predictive insights through modeling and AI. We can edit this app directly.
So we can go up here and edit the text. Again, visual editing is important aspect of this. Um, we can drag and drop these components. We can resize things. Um, the point is you can go back and forth between AI, visual, and code super seamlessly. And then if we need to, we can actually go directly down to the React code. So we can change the format of the icons that we had in the map component here. Save that. And then we see those changes propagated to the map.
Jumping up here, we can now let's flag this skew as critical. And so what this is going to do, it's going to add this to our shipment priority queue. If we click into this component, we can see that this is actually DBSQL with an AI query. And this AI query is using a foundation model to query unstructured shipment data and summarize the reasons the shipment is at risk. So we're processing unstructured data using this foundation model.
And finally, no demo would be complete without Genie. Um so we go back up here into this chat component and we can see a chat interface that is powered by Genie. So now we can ask questions of our data through natural language. And you can see this is pulling from the Genie conversation API. All right, the app looks good. Now we're going to deploy it on data bricks. So we're going to seamlessly deploy this as a data bricks app.
And what this is going to do is it's going to take us over to the data bricks workspace. And now we can see a running version of our data bicks app. Here we can see the data bricks app URL where this app has been provisioned. We can see the app resources that were attached to this app. So the SQL warehouse, the serving endpoint etc. We can actually edit these permissions. So if we go up here um we can see which resources have been added to the app.
So this is essentially adding the necessary production resources now that we're deploying this app to production in this workspace. Those look good. The next step here would be sharing this app um with the appropriate teams. So here we have this app preconfigured to share to my logistics team. So now they can access this app directly with O managed completely through data bricks. This all looks good. Now we can deploy this app.
We're pushing the new configuration and code to the container and we'll get a new URL with the production ready data bricks app. We can click on this and here we now have our app running completely in production as a data bricks app. data intelligence apps really helps accelerate democratization of data and II within your organization. No longer is this stuck behind technical interfaces and requires knowledge of coding or Python or SQL.
Um, anybody can tap into data and AI through applications. Data bicks makes it easy to deploy, secure and govern production applications and data bricks ops is ready and GA today and used across thousands of customers. I would encourage you to go use it. Thanks everybody. Back to you. Awesome. We're very excited about data bricks apps. Uh best way to connect to the data that you already have in data bricks and it's a great way to democratize access inside of your organization, right?
You don't need to give people access to all of data bricks. um it's just you know you give them just the app and then you have the authentication you have the lineage and all of the integrations just work out of the box okay so um as I mentioned there are three pillars to the future of all data intelligence applications that people will build we've been a lot focused on the database layer so far what we call the lake base and then we talked about how you can actually build your own apps on top of that because now you have your Postquest instance uh so now we're going to shift gears uh to AI for the rest of the day uh during the keynotes and as part of that um I want to welcome and talk to uh uh who's actually joining in uh from a friend's wedding.
So uh super excited to have him on. I hope the video link works. , uh thanks for having me. So this is awesome. We were talking about doing this conference and you were going to come here and then the you know and then there was a wedding. Uh I hope your friend doesn't know that uh you're you're you're doing this right now. he'll be very upset I'm sure. Uh so um super excited about the partnership. Um you know uh you told me that you know you've seen this kind of hockey stick uh adoption of entropic whenever you've launched it and he said we're going to see the same and I was a little bit skeptical.
I was like h you know I'm not sure it's actually I mean you know is it really going to take off like that and again u you're 100% right. I was wrong again and it's it's it's amazing to see. Uh so I'm curious um you know from all the use cases that we're seeing with enterprises using Enthropic together with data bricks um which ones you know are exciting to you and which ones kind of have maybe surprised you? Yeah. So uh maybe one that stands out to me is uh uh Block the company formerly known as Square.
Um so uh Block um made an internal AI coding agent uh that they call uh Goose. uh and as with uh as often happens with uh you know when when people use our models for coding this is a pattern we've seen you know at first people directly use the model for uh for coding and it it you know it received a lot of adoption there something like 75% of blocks uh blocks engineers but uh uh gradually it became a general tool for coordinating lots of different types of tool use lots of different data sources models that write code can write code for the user but they can also write code in order to use tools and take actions.
Um, and so blocks usage really expanded beyond the narrow coding use cases to many things the models can do with conia. And it's my understanding that goose is is you know has become incredibly popular within block and they feel that it's it's greatly increasing their productivity. I think something like 40% of not just their engineers but but you know of all the employees at the company are now uh are now using it. So you know that's really a a success story.
We're we're excited to get the productivity gains and for it to continue to grow. And I think it's also an example of, you know, something you've been talking about for this whole summit, which is, you know, the AI models are great alone, but but the best thing is when you can combine the AI models with the data sources with various tools, tools, and affordances where you can use the AI model as a as a as a as a brain that connects to various types of static information that you have in various dynamic frameworks.
So, you know, I think that kind of illustrates the thesis of our of our collaboration. That's awesome. Yeah, super excited about Goose. Actually, I used it personally u you know, your models are really good at coding, I have to say. And uh all of database engineering actually, you know, so we have over 2,000 people that now leverage goose. It's you know, one of our go-to platforms for uh you know, agentic uh coding. Uh, so I'm curious, Darus, you t you talked a lot about virtual collaborators are going to become common and we're going to have these agents, you know, working side by side us maybe doing, you know, multi-hour work.
Um, you know, I'm very curious that world. What will it look like? And also, how do we make sure that they're doing uh, you know, the right work? How do we evaluate them? Uh, and how do we make sure that they don't, uh, do something that, you know, would violate security or privacy? So, how do we like really govern them? Yeah, let's take those questions one by one. So I think first on on what it looks like, I guess we can talk in terms of coding and more more generally.
I think coding is moving the fastest. It's where we're seeing things first, but I think it's a little bit of a foreshadow of what's going to happen across all applications. So so within coding, you know, the thing I always refer people to is this post by called the revenge of the junior developer. And I can't lay it out better than he has. um uh and you know his his picture is you know we started with coding and you know first people just wrote code then there was the autocomplete er era now we're moving to what has called five coding um you know that has has come to be associated with coding models and claude claude in particular which is where you kind of ask the model to do something and it's it's very interactive I think then we're going to move to what ye calls um uh agent agent fleets uh which is where you dispatch a number of agents to do things for you and you act as essentially the manager of the agents.
Uh and then you know it'll go from agent fleets to agent swarms which is when each agent in the fleet itself deploys something and so the human engineer is sitting at top a hierarchy. It's like they're managing an organization or a or or or a company and they still need to intervene. They still need to set direction but it's a it's it's you know it's a much more leverage thing. Um uh so so in terms of governance I mean you know you can imagine if you have all these agents running around whether they're doing coding or whether they're doing something much more broad than coding which we expect with you know the kind of virtual collaborator agents that we expect to build that we expect to operate within within enterprises.
You know making sure that the that you know the agents don't you know vent private information to the internet. they don't take random actions that you know you're would give your security team a you know a heart attack. um these these are very important but I think fortunately you know in in many ways I wouldn't call these solved problems but these are problems we've been addressing for a long time right collectively as a community we have this idea of you know privileges and permission we have this idea of you know security requirements and security boundaries and you at data bricks you know this has been very very central for you you know governing the the you know the permissions and the safety of the model.
So our thinking is that we can slot our agents into it that all the things we've developed for data governance and governance of processes for humans, we can in a way sort of treat the agents the way we would treat humans and in other words make you know make the agents more powerful but also give them give them asurances. And so again this is an example of how we can we can kind of put these two things together and make something that's much greater than we can make with either component alone.
Yeah, I love that. And that was focused from day one when we started talking you know security governance how do we actually make that sure that's working as we uh launch cloud to you know all of our customers just seamlessly in the platform. Um so at databicks one thing we're really excited about is helping AI reason on the proprietary data that these organizations have you know basically domain specific intelligence or we call it data intelligence.
Uh I'm curious what are you seeing enterprises do with uh data intelligence and you know uh what's it going to look like going forward? Yeah. So I think this is an area that's accelerating quickly that's really resonating. I mean you know the ability to use data that's specific to an enterprise. You know it drives everything from an enterprises desire for fine-tuning to you know they enterprises see it I think correctly as you know a source of unique and competitive advantage.
So I've seen maybe three general patterns or general directions. One is you know enterprises and you know many enterprises will go to multiple of these use cases that want to make use of kind of the basic knowledge within their enterprise. So you know they want an agent that knows every piece of context about business and can help teammates quickly find it. So, you know, this is this is the thing that would have formerly been described as rag, but I think it's, you know, it's gone it's gone beyond that and there are many ways to do it and we're excited to work with data bricks on it.
Um, there's a second bucket I would call maybe automation of processes. So, you know, this this would be anything from, you know, kind of kind of classical RPA type tasks to to just automating any kind of process that's repetitive that happens within within the organization. And you here we want to make agents that kind of deeply understand business operations while again handling the self-governance making sure they don't go off the rails and are able to keep humans in the loop.
And again cloud models are good at this. Claude models specialize in that and the framework we can build around it with data bricks is also helpful. um the kind of create creativity research and analysis whether that's doing financial analysis or writing code or putting together a bunch of information into some kind of report. agents that act autonomously. This is what we were talking about in the last section. And you know, are kind of able to, you know, improve the business, you know, improve the quality of code, help with a new product release, a new marketing strategy.
Um, so, you know, I think we're increasingly leveraging these in in in in all the different areas, including to improve our own internal productivity, and we've noticed substantial improvements in our own internal productivity due to all three of these use cases for claude. I mean, you know, I think it's always good advice that if you yourself are not, you know, are not are not deriving benefit from something, you probably shouldn't be selling it to others.
So, we we are indeed driving benefit in this case. That's awesome. Yep. You should dog food your own stuff. Uh we do that every day. Uh that's super awesome. So, yeah, you mentioned cloud many many times. Uh so, you finally bumped the major version. Uh and we now have Opus 4, we have Sonnet 4. Uh models are amazing, especially at coding, but also, you know, writing, reasoning. Can you tell us a little bit about these, you know, tell us what's happening behind the scenes and, you know, what you guys did to make these models?
Yeah, so as always, I mean, you know, it's a mixture of uh, you know, it's a mixture of data, more compute, new training techniques, new, you know, new new new environments to train the models on, new algorithmic ideas. It's it's never one thing. It's always a mix of a variety of things. But, you know, I think a few things we're excited about. One is we're excited to have an Opus model. Again, um we haven't had an Opus model for a while.
They're, you know, they take longer, they're bigger, they take longer to train. Um but but uh you know, even I think beyond what's reflected in the benchmarks, Opus can do things that, you know, no other model and profit or otherwise is is is able to do particularly in the coding domain, but you know, beyond that. And I think, you know, Sonnet has been able to capture quite quite a lot of that um uh power while still being at the kind of affordable price and speed.
I think, you know, one thing I'm particularly excited about is, you know, the previous models had a little bit of a tendency to be overeager to reward hack. It was hard to get the balance right between models doing what you told them to do and the models being overeager and doing more than you would expect them to do. So just the fine sculpting of the models, following instructions, following human intent across coding, tool use, various other applications like customer service.
That's something that, you know, it takes a lot of iteration to get it right and we're still not perfect, but I think we we've improved the process quite substantially in Opus. Awesome. Okay, my last question. There's so much happening in the AI space. It's hard to keep up. There's innovations coming all the time. Um I'm curious is there anything that you've seen recently that you're excited about and you know that in particular is going to impact enterprises and how should they prepare for that?
Yeah. So, uh, you know, I would say the virtual collaborator, the virtual co-orker. This is something that we're working on. Are gradually introducing the pieces of I don't think there's any one place where there'll be one big release, but we're we're building the pieces of something that can, you know, use use all the interfaces that a human can um uh when when interacting with a with a computer and you know, behave in many ways like a a human a worker that can be deployed as an assistant by humans.
So you know I think again as with many things probably code will be the leading edge of it and we'll have these agent fleets that we describe but I think it'll go beyond that. I think that you know we've been working on computer use for a while. I think computer use is an area that's going to that's going to advance and again you know in collaboration with and data bricks I think we're we're going to be thinking very hard about the affordances that the models have the sources of data they can access and the permissions around those sources of data the tools they can use to connect to everything else on on the cloud I think that'll be an important part of providing the models context just as you know humans can be very smart but they you know their ability to log into their machines and you know install and use various tools, you know, in in many ways you're crippled without that.
So I'm, you know, I'm very excited for that general direction. That's awesome. Yeah, super excited about that as well. Looking forward to many many more years of partnership. , thanks for calling in. I'll let you get back to the wedding. Uh let's give it a round of applause for. Thank you so much. Thank you for having me. All right. So now we are uh we're done with the databases. We're now talking AI, right? As you noticed.

`1:28:45` Introducing Agent Bricks: Auto-Optimized Agents Using Your Data

Um so uh a year ago uh at this conference uh we actually announced uh the Agentic framework for you building your own agents that have data intelligence. So agents that can reason on your company's data, right? So artificial general intelligence can answer any question about anything on the web. That's super impressive. We leverage that. We leverage the models that Entropic provides. They're available on all the clouds.
You can just use them. Um, but how do we make the AI really reason on your data with high quality? So, that's why we were excited about the agent framework when we announced it last year. Um, we were excited about agents. Everybody's excited about agents. You heard about the uh virtual collaborators which are also agents in the organization that work with you. Uh, you can see the titles here, right? It's like doesn't not a day goes by without you seeing all these sort of the excitement around what's going on with agents.
But we spent the last 12 months talking to our customers, seeing what they are doing uh and are they actually getting these agents into production? What are the challenges they're seeing? What should we really do differently if we really want to make them successful? And you know um despite you know all the I would say a little bit hype around this um these agents could be built in a much simpler way. It's still really really challenging to build agents that are super reliable and just get the work done.
So why is that? We spent a lot of time talking to our customers to you uh and seeing you know where were the challenges where are you hitting roadblocks and why is this so difficult and we saw three things come up again and again and those are the ones that we're going to address today. So we're excited about that. So what are the three problems? So problem number one that we actually heard again and again is that I actually don't know how my agents are doing.
Like I don't know if they're doing high quality work or not, if they're doing errors, if they're hallucinating or not. In fact, we had a customer that had built uh agents that answered question about their product and this was a you know car manufacturer and when you asked it what's the best car I should buy, it was recommending the competitor's product. Okay. Um you know maybe the agent is really smart uh and maybe it's right. uh but you know our customer was not happy uh so you know that's one but sometimes they'll tell you also that some products don't exist even though they exist in the catalog or sometimes they hallucinate features and this is true for any agentic system out there I mean this is how these large language models are right um you know sometimes they hallucinate sometimes they get things wrong u the quality isn't yet there where we can just you know make it work so that's problem number one how do we make sure that we can evaluate them and in the industry largely everybody's super focused on artificial super intelligence or artificial general intelligence. uh and the focus has been on very difficult benchmarks like can you solve math olympiate level problems or can you do international coding competition uh for us and can you get what score do you get on that but you know we don't care about that we just want you know when I ask you about the car can you please recommend this car this is like the best car we're selling and please don't say we don't have this car we just launched it and so on and so forth so how do we get evals and benchmarks that really work on our data so that's problem number one and that's the main one that we've been focusing on at data bricks.
Problem number two is there's so many techniques coming out all the time. People are being inundated. I mean, as soon as they do something or they use some vendor or they use some technique, there's a new one coming out and you know what you did is you know so last week uh it's completely outdated. Why are you doing it that way? Have you not seen this latest tweet by? Uh so it just people are having a hard time keeping up with everything that's happening and they want to have you know uh state-of-the-art agents.
Uh so this is another thing that came up again and again and there's almost a fatigue around uh what's happening in the industry. Uh third one that we heard again and again and this was a little bit surprising is that they really want to balance cost versus quality. There's such a wide difference between the most expensive models and really really small tiny cheap models and it matters for the organization. Um, but you also want to have adequate quality.
So, how do you balance this trade-off of quality versus cost for your agents? So, that's why we're very excited. This is another one of our big announcements here at the conference, probably the last one today, uh, to introduce agent bricks. Okay, so these are production AI agent autooptimized on your data. Okay, so they provide you data intelligence and the way it works is that there are bricks for the different agents that you can build.
They're high level. They map to your business problems in your organization. So unlike our agent framework from a year ago, uh which was much more lowle, this maps to the business problems you have and then it uses agent framework under the hood. So how does agent bricks work? Um from a high level, you describe the highle problem that you're trying to solve. Okay, so build an agentic system that answer questions about automotives that we offer.
Okay, so you describe that and you know you can give it more description, you can be more prescriptive. So the more you describe the problem, the better it is. Then we do three things to address the three problems that I mentioned earlier. Number one and most importantly, we automatically create LLM judges for your specific problem. So no more, you know, math contest olympi level stuff. uh but instead you get an agent that basically can evaluate.
You can plug in any AI system that you have built and it will tell you this is the score you get uh if you want to have you know an agentic system that answers questions on automotives that we offer. Okay. And you get a score on that. Okay. Uh and this builds on the particular insight that these large language models are much better judges than creators. Uh it's intuitive. It's just like humans. Um, I can tell you if a painting is really beautiful and if a painting is ugly, that does not mean that I can paint the beautiful painting.
Okay? Uh, trust me. Uh, so it's the same thing with these AI models. Uh, they're the same way. Uh, so we'll help you build that. And then of course the judges could be wrong as well. So there's a way to have a human in the loop where the human can go in and uh, make modifications and say, "Well, this question doesn't make sense and this is not the answer you should be expecting." Once you have that, we have a way to evaluate your custom uh, problem.
This is essential. If we can't evaluate how the agent is doing at the particular task, there's no way we can do anything from there. So this is the key number one ingredient. Number two, we autooptimize the system. Remember I said there's like a zoo of techniques coming out. Well, we have hundreds of people. We have a dedicated research team uh of PhDs that are just working on what's the best way to get the best quality and you know the right cost for that particular problem.
Okay. And they've actually developed some really interesting techniques. uh but they also look at what's coming out and this is going to continue evolve right so we will optimize uh an AI system that does really well on the LLM uh judge benchmark okay so we'll try to get a high score on this automotive motive offering agent uh by looking at all the different techniques that are out there looking at all the different models you know there models from the frontier labs there's open source models you can fine-tune them you can do metarmpting you can use vector search so we'll do all of that for you automatically And then once we've done that uh we'll actually present to you uh a cost versus quality uh graph where you can literally see different dots that I can give you an AI system the quality will be this much but the cost will be that much or I can you know have a lower quality but significantly cheaper and this matters for some problems you know you can get way cheaper and only sacrifice quality a little bit and for some other problems you want to really pay up and it's the only way to get the quality to be good enough for your particular ular problem.
And then the most important part of this is that it loops and you can repeat this again and again. And our research team has developed a technique called agent learning that really helps you uh get feedback into the system and generalize it and keep optimizing it on new data uh that comes in so that the system gets smarter and smarter over time the more you use it. Okay. So that is what agent bricks is. Um I'm super excited to welcome on stage uh Hanin Tang who's CTO of neural networks at databicks to tell us about how agent bricks works under the hood.
Let's welcome. Thank you. I'm very excited to be here today to do a deeper dive on agent bricks and all things AI at data bricks. We spent the last year rethinking the best way to build agents and we're very excited to share that with you today. So here's how agent bricks works which allows you to build highquality and production ready agents on your tasks. First you select a task and then declare a highle description of what you want the agent to do and then you attach your relevant data.
From there agent bricks solves the three problems that the just spoke about based on your data and your task. We automatically create quality evaluation benchmarks. Evaluations are very specific to what you want your agent to do. For example, when the task is to summarize customer support transcripts, we'll create very specific judges such as customer churn. Now you can be confident about measuring agent quality and bring it to production.
This is actually quite incredible. This is one of the few products out there that'll automatically create evaluations very specific to what you're trying to do. Then to deal with what mentioned about so many different techniques that are out there, agent bricks under the hood will automatically search through and compose different optimization methods and settings to deliver high quality. We may try fine-tuning, prompt optimization, reinforcement learning or agentic workflows and patterns.
So now you can trust that all the knobs are already optimized and you're sitting on the cutting edge of research with agent bricks. This then allows you to build agent systems that are both higher quality and lower cost compared to say simply DIY prompting a large language model. Then you can deploy and iterate on quality. Today we have four tasks with more on the way. Information extraction which is an optimized pipeline to extract insights from documents.
Knowledge assistant to build high quality question and answering or conversational agents on knowledge sources such as documents multi- aent supervisor which is probably my favorite which is able to stitch together multiple agents MCP servers and genie agents into a single system to tackle very complex reasoning tasks and lastly we have custom LLM which able to support custom tasks such as content generation customer that answers I don't know technical support questions and telecommunications and attach your knowledge sources.
Then we go figure out the exact agent architecture to and optimize the configuration for quality. This is how agent bricks works.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.

Data + AI Summit Keynote Day 1

Summary

Data + AI Summit Keynote Day 1

Chapters

FAQs

What is Databricks Free Edition and who can use it?

What is Lakebase and how does it support data apps and AI agents?

What are Databricks Apps and what can they do?

What is Agent Bricks and how does it help developers?

Full transcript

0:00 <Untitled Chapter 1>

20:30 Introducing Databricks Free Edition – Learn and explore with data and AI for free on the same platform used by millions of professionals

27:30 Announcing Lakebase Public Preview – Fully-managed Postgres for data apps and AI agents

46:40 Lakebase Demo: Holly Smith – Lakebase

58:30 Announcing General Availability of Databricks Apps – Build Secure and Governed Data Intelligence Apps

1:09:15 Demo: Justin Debrabant – Databricks Apps

1:28:45 Introducing Agent Bricks: Auto-Optimized Agents Using Your Data

`0:00` <Untitled Chapter 1>

`20:30` Introducing Databricks Free Edition – Learn and explore with data and AI for free on the same platform used by millions of professionals

`27:30` Announcing Lakebase Public Preview – Fully-managed Postgres for data apps and AI agents

`46:40` Lakebase Demo: Holly Smith – Lakebase

`58:30` Announcing General Availability of Databricks Apps – Build Secure and Governed Data Intelligence Apps

`1:09:15` Demo: Justin Debrabant – Databricks Apps

`1:28:45` Introducing Agent Bricks: Auto-Optimized Agents Using Your Data