ADBC and the Future of Universal Data Connectivity for Agentic Systems

Summary

ADBC (Arrow Database Connectivity) is a vendor-neutral API and driver standard built on Apache Arrow's columnar format that replaces slow, row-oriented ODBC and JDBC connectors with fast, direct columnar data access for agents, pipelines, and analytics tools.
Legacy connectors like ODBC (1992) and JDBC were designed for row-oriented databases and downstream applications, making them fundamentally mismatched for modern columnar data systems and the tight latency and cost budgets of agentic tool calls.
A live demo shows Codex and Claude using an ADBC driver to query Databricks directly, illustrating how lean, CLI-first building blocks outperform framework-heavy integrations for agentic data access workflows.

ADBC and the Future of Universal Data Connectivity for Agentic Systems

Agents are becoming first-class users of the data stack, but they inherit 1990s-era connectivity: slow row-oriented APIs, heavy SDKs, and brittle system-specific connectors. For AI-driven workflows making fast tool calls across systems, this isn’t just slow—it’s terrible agent ergonomics, leading to failures, retries, and need for human intervention. This talk explains why ADBC (Arrow Database Connectivity) is emerging as the universal connector for agentic data workflows: a vendor-neutral API and driver standard that ships Arrow columnar results, enabling fast data access across languages and engines. We’ll show why this matters for agents: frequent queries, tight latency/cost budgets, and the need for safe, reliable access to the tabular truth. You’ll leave with a clear model for where ADBC fits in lakehouse architectures, why it aligns with Databricks and open standards, and a how to move from framework-heavy integrations to lean, CLI-first building blocks guided by agent skills.
Talk By: Ian Cook, Co-Founder and CEO, Columnar ;

Chapters

FAQs

What is ADBC and how does it differ from ODBC and JDBC?

ADBC, or Arrow Database Connectivity, is a vendor-neutral API and driver standard that ships query results in Apache Arrow's columnar format rather than the row-oriented format used by ODBC and JDBC. This matters because modern data warehouses and lakehouses store data in columnar formats, and converting columnar data to rows to send through ODBC—then converting back—wastes significant time and memory.

Why do legacy database connectors remain so common despite their inefficiency?

ODBC was created in 1992 when all mainstream databases were row-oriented, making it a natural fit at the time. The standard became so widely adopted that an enormous ecosystem of tools was built around it, creating inertia that has persisted even as query engines, storage systems, and downstream applications moved to columnar formats over the following decades.

How does ADBC benefit AI agents specifically?

AI agents make frequent, latency-sensitive tool calls to query data, and the row-by-row retrieval pattern of legacy connectors creates compounding inefficiency at this scale. ADBC delivers columnar results directly in the format that DataFrames and embedding pipelines expect, reducing conversion overhead, cutting latency, and making agent data access more reliable.

What languages and tools does ADBC support?

This video demonstrates ADBC with Python including Polars via DataBow, DuckDB for in-process analytics, and a Databricks-specific ADBC driver. ADBC also supports JavaScript and TypeScript, making it available for multi-language agent architectures that need consistent columnar data access across the stack.

Full transcript

[00:07] All right, welcome everyone. Uh so, I'm Ian Cook. I am the co-founder and CEO of Columnar. Um and at Columnar, we are building open-source infrastructure to make database connectivity much faster and simpler. So, this talk is about um kind of why
[00:21] the era of AI and agentic systems has made this mission um a lot more important and more urgent than it was before. Uh but before I talk about ADBC, uh I want to talk about a familiar experience
[00:35] I think many of us have probably had. So, raise your hand if you've ever pulled a large query result out of a data warehouse, data lakehouse into another tool. Okay. Now, keep your hand up if it arrived
[00:47] instantly and exactly the format you wanted with no extra copying, no conversions, no driver issues, no memory issues. Right. Nobody. So, um database connectors tend to be a big source of
[01:01] disappointment. Like, often um they're so bad that it seems like a weird anomaly cuz everything else in the ecosystem has gotten so much better. Uh query engines, for example, have gotten, you know, incredibly fast. Like, just
[01:13] look at some of the keynotes uh this week. Um storage has gotten faster, data transformation systems, analytics systems have moved forward, but this connector layer just often feels really old and slow. Um and actually, there's a
[01:25] pretty good historical reason that it's been so bad for so long. So, we're going to take a little journey back to the '90s uh for a few minutes. Um back then, the whole data world looked quite
[01:37] different. Um all mainstream databases in the '90s were row-oriented. Uh and also, most downstream applications also expected data in row-oriented formats. Um and the problem
[01:51] the industry faced like back in the early '90s wasn't speed, it wasn't ease of use, it was just like basic interoperability. Like how do we connect downstream tools to SQL databases in a standard way?
[02:03] So, in 1992, a group of companies got together led by Microsoft and Simba to solve this problem, and the result was ODBC. Which, of course, stands for open
[02:16] database connectivity. And at the time, in 1992, this was great, right? It unified a bunch of previously incompatible systems, which made users happy, allowed the whole ecosystem to grow and flourish. Later on in the '90s,
[02:29] the folks at Sun Microsystems decided they wanted their own Java-specific standard, so they created JDBC. A little bit later in the '90s, Python introduced their own data connectivity
[02:41] API, DB-API 2.0. And, you know, because the systems that these standards were connecting were all row-oriented, ODBC and JDBC and DB-API, of course, were also row-oriented.
[02:54] So, these standards and a few others like them became pretty widely adopted at the time. And today, you know, more than a quarter century later, right? These are still familiar names. Like we still we still use these.
[03:06] And what is pretty surprising about that is that since the '90s, the whole rest of the data world has changed pretty dramatically. Like one of the biggest changes was the rise of column-oriented
[03:18] data systems. So, starting as early as like the '70s, there was some academic research into this that, you know, kind of concluded that if you organize data by column instead of by row, it makes most
[03:31] analytic queries much faster, much more efficient. So, database researchers sometimes use this term DSM to refer to this columnar storage approach. I'm not going to get into like the technical details, but this is a slide from a
[03:43] Carnegie Mellon database seminar that talks all about this. The basic intuition, if you're not familiar, for why columnar systems are faster at analytics is that database tables
[03:56] typically have many columns, but most queries usually only need to like filter or process a few of those columns at a time. Um and computers can just get that done much much faster if they don't have to constantly jump around, scan through
[04:09] loads of data they don't need, stuff like that. So, columnar architecture avoids that. Um and just as one example, you know, the Databricks Photon engine uh is faster than open source Spark and one of the reasons why is because it
[04:21] uses uh columnar processing. So, because because of these speed advantages, by the 2010s, like almost every major analytic database system uh used columnar architecture.
[04:35] But unfortunately, the connectors did not follow suit. So, uh you know, by that point, ODBC and JDBC and all these other like row-oriented connectivity standards were just very deeply ingrained. They were
[04:48] integrated, deployed everywhere. Whole big companies were created just to build and sell drivers. Um so, yeah, kind of like the row-oriented toothpaste was out of the tube, smeared all over everything. Uh you can't put it back in. So,
[05:01] even as these slow row-oriented connectors became a bottleneck, um kind of this pinch point in the middle of the, you know, the columnar ecosystem, uh we kept using them uh because of inertia, because of network effects.
[05:14] Also, by that time, the whole like data analytics industry had grown pretty large and the thought of coordinating all the players to agree on a new better standard was uh
[05:26] seemed pretty daunting and unlikely. So, I have some visuals here to help uh make it clear why it's so slow to use a row-oriented connector in a
[05:39] column-oriented world. So, here is a simple database table. Um three columns, ABC, six rows, uh and this is how we think of a table, right? In this two-dimensional presentation.
[05:51] But, in computer physical computer memory and computer storage, uh everything is one-dimensional. So, we need to serialize the data and put it in a one-dimensional series. So, like just
[06:03] for background, if you're not super familiar with this, computer memory and storage is all laid out like that. It's just like one long one-dimensional sequence of bytes. Um it's kind of like reel-to-reel tape in computers from the '60s, right? One
[06:15] long strip of data. Obviously, on computers today, you can jump around a lot more efficiently and quickly than you could with uh you know, reel-to-reel tape. Um but still, it's less efficient jumping around than it is just move through the data sequentially.
[06:29] So, if you serialize this table in a row-oriented layout, it looks like this. So, you can see like first, the column names and other schema information, and then the first row, all the values, and the second row, all the values, and so
[06:41] forth. But, if you serialize the data in a columnar layout, it yeah, it looks different. Um another a small detail kind of just worth saying about columnar formats is that uh usually what we do is
[06:55] we split the data the whole table into chunks or batches, and then serialize those batches column by column. So, these batches are often pretty large, like a million rows. Uh in this simple example, I'm using
[07:07] three-row batches just for illustration. So, we've got, you know, the schema at the beginning, and then we've got all the values from the first batch, but each um of the column values uh is adjacent. So, column A values
[07:19] here, column B values, column C, and then the second batch, and so on. So, let's see what happens when you move data from a columnar source format, which most things are these days,
[07:31] through a row-oriented uh connector into a columnar destination format. So, like for example, this is what happens if you pull data from uh
[07:43] from Databricks through ODBC or JDBC into Tableau. Or, you know, there's a million examples from Snowflake through Python DB API into Pandas, which both of, you know, Pandas and Tableau are both columnar internally.
[07:57] So, you need to take the columnar source data, and you need to jump around and rearrange it all into a row-oriented format. And this is kind of slow.
[08:12] All this jumping around. Okay, so, um now you can pass this data in a row-oriented format through the connector. This is what, you know, ADBC, or sorry, ODBC, JDBC, DB API all require. Um but, the destination system wants it
[08:24] in the original columnar format. So, now we need to reverse all that work. Jumping around, rearranging it back into the columnar format that it started in. Um and all this jumping around takes a ton of CPU cycles, and it actually makes
[08:38] data transfer CPU-bound instead of network-bound. Um and besides wasting a lot of time, it also wastes a lot of energy. So, if that all seems a bit ridiculous,
[08:50] um yeah, it is. I mean, it's actually even worse than that, because uh even beyond like the fact that they're row-oriented, a lot of these connector APIs add in some other 90s jank, uh you know, kind of above and beyond um what I
[09:03] described here, like poor compression choices and so forth. So, if you've ever wondered why, you know, tools like Tableau uh can be notoriously slow at pulling large query results from data
[09:15] warehouses, you know, this is why. Uh okay. So, um yeah, for the most part, like we've just been doing it this way for 30 years because we can't agree on something better.
[09:29] All right. Why is this not working? All right. So, database nerds actually have a funny name for this kind of process where the data is forced to be handled one row at a time. Um they call it RBAR, which is row by agonizing row. I also found out
[09:41] that RBAR, somebody told me is a dive bar in San Francisco. Has anyone actually been there? I I might have to go there. I don't know. Um So, you know, what would be kind of obvious, right? Uh is if we could just
[09:54] keep the data in a columnar format all the way through. Uh no jumping around necessary. Just sequential bulk data movement. Goes really fast. Doesn't tax your CPU. Um makes sense.
[10:07] Uh I'll get to that in a minute. But, another interesting thing that's worth saying uh is that since the late '90s uh networks have gotten really fast. So, while CPU performance has increased,
[10:19] right? By about 100x if you measure it in flops, network performance, network speed has increased by more like 10,000x if you measure in bits per second. Um and so, what this means is that faster hardware hasn't solved this problem for
[10:33] us because transferring data through these like inefficient row-based uh formats actually gets more CPU bound with every passing year because of this discrepancy of networks getting faster than CPUs are.
[10:45] Um Yeah. Okay, back to the history uh cuz this is kind of the point at which the plot starts to turn around. So, in 2017 at a database conference called VLDB, these
[10:59] two guys um Mark Raasveldt and Hannes Mühleisen from CWI in Amsterdam wrote a paper all about this problem of slow database connectors. Did any- anyone recognize these names? Well, you guys do. Yeah.
[11:12] They're they're the DuckDB creators, right? They they hadn't created DuckDB yet. Um they didn't start until about a year after publishing this paper. But, this paper focuses sort of not on the connector APIs like ODBC or JDBC
[11:25] exactly, but more on the the client-server protocols um that those connectors use to talk to databases over the network. Uh and their main like for example the Postgres protocol or the MySQL protocol.
[11:38] So their main finding is that uh these are mostly very inefficient and it's mostly because they iterate row by agonizing row, uh but also because they make some poor choices about things like compression and encodings. Um this is a
[11:51] really good paper uh and it proposed some alternatives that end up looking a lot like what we'll talk about next. So around the same time that Mark and Hannes were working on that paper,
[12:03] unknown to them, um a group of open source developers was working on what would eventually become a good standard solution to this problem. So Wes McKinney, who is the creator of Pandas,
[12:15] and Jacques Nadeau, who is one of the co-founders of Dremio, uh got together a group people to create the Apache Arrow open source project. And the main goal of Arrow was to define a standard
[12:28] columnar data format for holding tabular data in memory. Um so if you remember like this picture from earlier, that is exactly what Arrow created, a like fully standard, fully
[12:40] well-specified, cross-language way of organizing data like that, columnar batches, um in memory. So the initial motivations for doing this were one, the Parquet file format,
[12:55] which had been created like about 3 years earlier, was becoming very popular, uh but it couldn't really achieve all of its goals without a complementary in-memory format, like a standard way to hold data from Parquet
[13:07] files in memory. Um and then another motivation was that uh systems like Apache Spark, which was also becoming very popular at the time, um they were built in the JVM and you know, Java, Scala, JVM languages,
[13:21] uh and they had no efficient way to transfer tabular data in memory to non-JVM languages like Python and R. So, anyone remember using PySpark in like
[13:33] the early days, like before 2019? It was really slow because it didn't have this yet. Um So, Arrow took off pretty quickly. Today, it's basically everywhere. Um there's not really like a single major
[13:46] data platform that doesn't use Arrow somewhere. Um so, I'm part of the core Arrow team, one of the PMC members. Um as Arrow grew, it kind of expanded from
[13:58] its original goal of exchanging data in memory on one computer um and expanded to exchanging data across networks. Uh so, it turns out Arrow is really good for that, too. Um especially when all the source
[14:11] systems and all the destination systems out there kind of use Arrow, uh you can avoid serialization everywhere by using it on the network as well. So, there are two projects that uh as part of the Arrow open source project
[14:23] that launched within 2022-2023 to solve this. The first one was Arrow Flight SQL. Um this is a database client-server protocol that keeps data in Arrow format
[14:37] as it goes uh over over the network from from the database to the to the application. So, technically speaking, like this Flight SQL kind of solved the problem that Hannes and Mark described, uh but
[14:50] there were kind of two problems within this. Like one was that in order to work, Flight SQL depended on the database vendors adding support for it. Um but early on, like
[15:03] Dremio was pretty much the only vendor that did this. Over time, a few a few more did, but this was a a real problem getting database vendors on board with this. And then the second problem is that it didn't standardize the very last mile,
[15:16] like the client API. So, in this original this is like the original diagram that was in the blog post announcing flight sequel and there's this sort of conspicuous absence here where you you see JDBC, you see ODBC and
[15:28] then you see this box that's like arrow native app. Like what does that mean? Like there's no standard there. We didn't have a great answer for that. Uh so those problems gave rise just a few months later to ADBC, uh arrow
[15:43] database connectivity, which solved both of those problems. So ADBC is a client-side data connectivity API. So as the name suggests, it's a kind of a successor to ODBC and JDBC. It's a similar style of API but much more
[15:56] modern, much faster, and it delivers data across the last mile um to the application uh and it uses arrow format to do that. So basically I mean the basically how it works as an API is you give it a SQL query
[16:08] and it gives you back the result as a stream of arrow data. Um and importantly, it does not strictly require the database vendors to add support for it. So like under the hood
[16:21] uh ADBC can speak any client-server protocol including ones that do use arrow and ones that don't use arrow. So ADBC is this is kind of like what's depicted on the right side of this diagram with using arrow on the top, not
[16:35] using arrow on the bottom. Um ADBC is much faster when the database supports flight sequel or has some other way of sending arrow data directly uh through the client, you know, through some
[16:47] client-server protocol but it doesn't strictly depend on that. We can make it work with anything like the Postgres protocol, the MySQL protocol, etc. Um and that has actually been like really important for unblocking progress because we can't
[16:59] just like wait for the vendors to adopt uh something on their service side to get moving with this. So the rest of this talk will focus on ADBC um and I want to just be like clear up
[17:12] front about what ADBC is and what it's not. It's a client-side API. Uh it's cross-platform. It runs on macOS, Windows, Linux. With a bit of work, you can get it to run on any OS.
[17:24] It's cross-language. It's not tied to any one language, any one runtime ecosystem, anything like that. I'll talk more about that later. It's vendor neutral. Databricks doesn't own it. Snowflake doesn't own it.
[17:36] Yeah, no no one vendor controls it. It's governed by the Apache Software Foundation, just the same way that like Parquet and Iceberg and Spark are. And it's open source.
[17:49] And it's part of the Apache Arrow project, and it kind of composes seamlessly with all of the other systems that speak Arrow. So now, just to clear up a few kind of
[18:01] common misconceptions about ADBC, it is not a client-server protocol. Remember, Flight SQL is a client-server protocol. ADBC is not. It's just a client API. And it's also not a
[18:13] a SQL transpiler. Like it doesn't automatically translate SQL dialects. Like you can use it together with SQL transpilers like SQL Glot, but it doesn't it's not going to automatically translate SQL for you.
[18:30] I want to run through three important terms that we use when we're talking about ADBC. So one is the ADBC API. This is a standard set of programming interfaces in several different languages for interacting with ADBC. The
[18:43] core API is defined in a C header file, but there are also APIs in a bunch of other languages like Go, Rust, JavaScript, TypeScript, Python, Ruby, R, and so forth. More coming.
[18:55] Those language APIs kind of stick to the core API, but then also are designed to follow idioms of those languages. So like for example, the ADBC Java API is
[19:07] designed to feel familiar to JDBC users. And the Python API mimics DB-API as much as it can while being column-oriented. Um, okay.
[19:19] Second important term here is driver. So, the thing that actually allows you to connect to a database with ADBC is called the driver. This is just like with ODBC and JDBC. Um, ADBC drivers can
[19:32] be implemented in any like low-level systems language. Uh, C++, Go, and Rust are the common choices. I personally kind of want them all to be written in Rust, sorry Matt. Um, but you know, you have you have some choice.
[19:44] Uh, ADBC drivers can be they can in some cases be like statically linked into an application, but not usually. Like you the usual pattern is that you build a driver into a shared library
[19:57] and then you dynamically link that shared library into an application at runtime. So, if you're if you're not familiar with shared libraries, this is like a .so file for Linux, a .dylib file for macOS, .dll for
[20:10] Windows. This is like a dynamic cross-language library. All right, last term here is driver manager. So, because ADBC drivers are typically not
[20:22] language-specific, you cannot just like import them directly into Python or into JavaScript or things like that. You need another library to do that for you, and that is called a driver manager. So, you import that the driver manager
[20:35] into your language, the driver manager then dynamically loads the driver. So, we have driver managers for most every mainstream language. All right, so this is a kind of a a snapshot of the state of ADBC today.
[20:50] There's a bottom growing collection of drivers that let you connect to uh, like, you know, dozens of different databases. Uh, we're releasing more drivers every month. Just uh, just this week we announced a Spark driver, which I'll I'll demo in a few minutes. Last
[21:03] week we announced a Data Fusion driver. So, they're coming fast and furious. Um as I mentioned, there are uh ADBC driver managers for almost every mainstream language. Uh ADBC is also directly
[21:16] supported in some data frame libraries, including Pandas and Polars. And we're also seeing more and more tools that are choosing ADBC as their standard connector interface. Um so this year, Microsoft announced that
[21:29] Power BI is going to standardize all their connectors on ADBC. Um and then just a few weeks ago, DBT announced that starting with uh version two of DBT Core, every adapter will be
[21:42] built on top of an ADBC driver. And there's like tons of integrations that I didn't even show here. Airflow, DLT, Bruin, Spice AI, RisingWave, lots lots more. Um so yeah, I mean you know, as
[21:54] incomparable as sorry, as improbable as it seemed just a couple of years ago um that we could overcome the inertia of ODBC and JDBC, we've kind of now managed to do it, but I'll say there is still a pretty long tail of integration work
[22:07] remaining, especially with kind of other downstream tools, data transformation tools, analytics tools. There's a lot of work needed to migrate um legacy connectors to ADBC there. Okay, so uh I'm going to get into demos.
[22:20] Um if you want to try ADBC, like how how can you get started? Well, step one is to install DBC, which is the package manager for ADBC drivers. So this is like just a simple
[22:32] open source command line tool. Um if you're familiar with UV for Python, it's heavily inspired by UV. Runs anywhere, makes it super simple to install drivers with a one-line command. The drivers are pre-built for all the
[22:44] operating systems, signed, notarized, so they just work. Um so I'll demo it and you can check it out at calmer.tech/dbc. Uh if you have questions, you can ask Matt who's here, who's the uh lead developer
[22:56] uh of of DBC. Uh another tool that's totally optional, but makes it really easy to get started with ADBC is Databou. This is kind of a universal query CLI that can
[23:09] connect to any database that has an ADBC driver. You can check this out at databou.sh. And if you have questions about this, you can ask Emil, who's the lead developer, who's here.
[23:23] All right, so I'm going to show a demo of using both of these tools. Um and this is going to be slightly janky because I'm trying to going to try to I don't know. Yeah, sorry. We had some tech problems. So, let me see how I can how I can do this.
[23:36] One moment. Yeah. I tried that before and it didn't work, but
[23:49] um I think I can just I'll just uh I'll just mirror. Yeah, that'll that'll do it. Uh okay.
[24:01] Oh, is it that actually going to work? Okay, well, if so, that's amazing.
[24:33] Uh I can just drive it from here if need be. No worries. Okay, um All right. I got my notes here. All right, so the way that you can
[24:45] install DBC and Databou is with UV UV tool install um DBC uh or Databou. I actually already did this, so I'm going to skip it. See if I
[24:57] can make this a bit larger. Um yeah, UV UV tool install. And then um once you uh have DBC installed, you can use it to install drivers. So, let me go ahead and install
[25:09] the driver for Spark. DBC install Spark. The driver is a pre-release driver, so I'm going to put this {{}dash} {{}dash} pre. I already installed it cuz I was afraid the Wi-Fi was going to be broken.
[25:22] Um but uh that is uh that That's how you install the driver. Uh okay, now I'm going to connect to uh this driver with DataBow, and I'm going to have to pass in a URI to um an instance of Spark that's
[25:35] running in a Docker container on my machine. So, with DataBow, you just tell it what driver you want it to use, and then if you need to a URI or some other parameters to connect, you pass those right on the command line. Um so, now we're connected to Spark. Um
[25:48] and I have a little like simple data set. I'm going to in these demos just use some toy data set. Um uh to show how it works, and we pull this data from Spark um and then
[26:01] displayed it here. So, the cool thing is that if you want to connect to other systems, it's exactly the same interface. So, uh I'm going to do DBC install Databricks uh so I can connect directly
[26:14] to Databricks. It's already installed, and then I can say DataBow. Um and for Databricks, I uh have uh a secret, you know, uh a like a token that I need to connect to
[26:26] it. And so, instead of typing that, so you can all see my secret, I've put it in a profile, a connection profile, which is ADBC's way of storing connection details. Um and it's called Databricks, so I can just do that. And
[26:40] then it's going to slow Wi-Fi here. It's going to take a second. Um but this will connect to to Databricks and will allow me to run run queries uh on Databricks. And you can do exactly the same thing for like any of the um you know, 20 or
[26:52] so drivers that exist which support like dozens of different of different database systems and then I can do the same thing. I have the same data set loaded in in data bricks so we got we got the same result
[27:06] and you know, you can keep going. So if you want to use DuckDB one of the cool things about DuckDB is we I skipped over the step of actually installing the DuckDB driver.
[27:20] But with DuckDB you can run select queries on any parquet file or other file that you have locally. So I have that same table in a parquet file locally and apparently I forgot to
[27:33] put it in this directory but it would work if it was here. So yeah, you can you know, connect to any of the systems that you want to see the full list of drivers that are available including the pre-release ones you can type this like DBC search
[27:46] command and it lists lists all of the drivers. We got quite a quite a big collection available. All right. Go back to the slides for a moment.
[28:04] Okay, so um I want to skip now to a couple of demos that highlight how ADBC actually works great when you're doing agentic coding with Claude with Codex and so forth. So there's a few properties of ADBC that
[28:16] make it like really effective for this. One is that installing drivers is super simple. One line CLI command any agent can do it. There's no searching around your you know, your disk for the right
[28:29] DLL file. It just works. The driver can the agent can do it. I'll talk more about this later but it minimizes the overheads of fetching data in multi-step workflows.
[28:41] Another important thing about ADBC compared to many other ways of fetching data is that allows you to be pretty confident about what the types of the data that you're receiving are. So, Arrow is a very strongly typed data
[28:53] format, um and so you know exactly what you're getting back. There's no uncertainty about whether, you know, the string will parse as a date or as an integer or things like that. Uh and also one consistent API across
[29:05] many different databases means that when your agent writes code that works on Spark, that code should work on Databricks, that code should work on Snowflake, it'll work across the whole the whole world of of systems that have ADBC drivers. And then similarly, um it
[29:20] allows flexibility about what language you're using. So, if you create a workflow that uses ADBC in Python, but then you're deploying it and you decide, "Ah, we need to rewrite it in Rust." No problem, you can use exactly the same drivers uh and a an API that is, you
[29:34] know, almost identical, but just in the Rust language instead of in Python. So, this all makes um ADBC quite uh quite effective. Um and then we also have a set of agent skills that reduce trial
[29:46] and error for any kind of uh any kind of agentic work you're doing with ADBC. So, I'll do a quick demonstration of this, um but one thing I'll do first is I'll show you that we have this set of agent skills. Uh this is the standard way to
[29:58] install it. There's different ways of installing agent skills, but this Vercel library and uh uh on NPM called skills is the is the usual thing people use. Um so, I'll I'll start by installing this and then
[30:11] I will um run some kind of agentic demos using that. If I can get my screen to cooperate. Mirror. Okay.
[30:34] >> All right, let me make this a little bit bigger. Okay. Um so, start off with NPX skills install
[30:49] All right, and this in this installs this the the set of agent skills for ADBC and some related tools from this Git repo. So, there's three of them here. One is for ADBC uh sort of in general, one is for this Data Boat tool I demonstrated, and one is for the driver installer DBC. Uh you can select
[31:03] which agents you want to use. You can select, you know, where you want to install it and so forth. I'll proceed. All right. Um and then I can start a um uh uh uh
[31:15] agentic coding session here and get started. So, I have a couple prompts. I'm just going to copy and paste um in the interest of time. Uh but what I'll ask ADBC to do uh what
[31:27] I'll ask Codex to do using ADBC is connect to Databricks, tell me how many stock ticker symbols are included in this table I have in Databricks, what range of dates it covers, um and I want it to use all the connection details and
[31:41] secrets I have in this connection profile called Databricks. Um so, I'm going to just let that go for a minute in YOLO mode. Hope it doesn't wreck my computer. Um fingers crossed. Uh and just while we do
[31:53] that, I'm going to show you a couple of other things uh about ADBC that are good to know. One, if you're looking for where um drivers are documented and how to find
[32:05] them, we uh have this project called the ADBC driver foundry. It's basically a centralized home for all open source ADBC drivers. It has documentation, it has guides, it uh has
[32:17] links to all the all the tools you need to use and install these things. So, for example, the Databricks driver is documented here. Um it shows how to install it. It shows how to use it in Python. It has um description of all of the the connection string uh format that
[32:31] it can take, the different auth options, and so forth. And then, for any of these drivers, you can click to see um the underlying Git repo uh just by, you know, going going from there. So, this
[32:43] is an open-source driver that's been developed by Calmer in collaboration with the team at Databricks. Um it's still a work in progress, uh but there's a bunch of a bunch of work going on here. Okay, let's see what uh is happening
[32:56] over here. Okay, so it got an answer. It said this data uh in this this ticker stock ticker from sample uh stock ticker table has 93 15 distinct symbols covering dates from 1962 to 2024. Okay,
[33:09] cool. So, 34 million rows total. Um So, that was that was good. But, how did it how did it do it? So, it actually used DataBow uh on the on the command line to do this. Um But, now I'm going to ask it like do
[33:22] this with Python instead of DataBow. Uh and I'll see see how that does. Uh while that's while that's cooking, I'll show you a couple other things. We have a uh cookbooks website that has
[33:36] a lot of like sort of task-based guides about how to do certain things with uh with ADBC. One uh that I recommend taking a look at is this one all about how to use connection profiles. These
[33:49] are these uh standard files that can contain all of your connection details, passwords, secrets, and so forth. Um there's an example of one here. It's just a very simple TOML file
[34:02] format that it lives in a specific place on your disk um that, you know, can include uh can include all of the connection details. It also has a way to, you know, uh keep passwords in environment variables so that you're not like
[34:14] leaking them into into into files like this. Okay. Uh So, looks like it got the same thing done with Python. Um I could ask it to do it in
[34:26] TypeScript. I could ask it to do it in a different language. But, let me actually go in a different direction here and uh I'm going to ask it not to do use data bricks but instead to use neon. So I've loaded in this exact same table in neon
[34:38] database. And I have a driver for post grass installed here. So neon database speaks post grass protocol. There is an ODBC driver for post grass protocol. So this
[34:51] should also work because I've pre-loaded the exact same data there. Again let that cook for a minute. And I will show another a great place to go again for for
[35:04] looking at this DBC driver installer. When you you go to this page columnar.tech/dbc there is this kind of wizard interface here that makes it really easy to get started. So if you're using Windows
[35:16] and say you want to use JavaScript or TypeScript as your as your language here and you want to connect to Redshift say. This provides step-by-step instructions. Here are all the installation methods
[35:29] for the DBC driver installer that work on Windows. Then here's how to install the driver. Here's how to install the driver manager for JavaScript or TypeScript and then here's how to like load that driver
[35:44] in you know in your JavaScript application. This also links out to a great resource called the ODBC quick starts which is just like a massive collection
[35:56] of examples covering I think eight or eight or seven different languages and just dozens of different source systems all of which are covered by like the 20 or so drivers that exist. So big treasure trove of information
[36:09] that can help you know get started or you point your agent at this if it's having trouble and it can usually pick things out really fast. For some reason it's still trying to figure out how to get to how to get to neon.
[36:22] But uh I'll I'll I'll that go in the background while I show you one more thing. Oh, wait. Did it finish? All right. Yeah. So, it got done
[36:35] and it gave the same result. So, the ability to jump between languages and jump between databases just seamlessly without having you as a human or your agent think about what the different interfaces are is just like an incredible and it's really a huge, you
[36:49] know, sort of on top of the speed of ADBC, there's also just this developer productivity and UX benefit as well. Okay. Um So, real quickly,
[37:01] uh I don't have time to do a full demo of this, of course, but um uh you know, it's not what I demonstrated just there is kind of fun little scripting tasks. But, what about like entire agentic app development?
[37:13] Well, ADBC is great for that, too. So, just yesterday, I, you know, open Claude code and give it a prompt like this. Basically, told it to build like a Yahoo Finance style stock ticker chart. Um
[37:26] using that same data that I was just querying uh and using, you know, sort of ADBC to push down queries as you pan and zoom around. Um not preload all the data at once. So,
[37:39] uh I let that go. I had to like fiddle around with it, you know, and uh give it some subsequent um further instructions, uh but it worked out pretty well. Uh and this is what this is what Claude whipped up after a while. And so, it
[37:51] just got this menu where you can select stock prices. Uh again, not as fast as I would love because it's connected on conference Wi-Fi here to Databricks. Um but still pretty fast. Uh in order to
[38:04] overcome the problem of the conference Wi-Fi, I also had it um uh sorry. Uh uh create a second mode that connects to a local Apache Data Fusion database
[38:17] querying a local parquet file. And this kind of, you know, simulates if you have just an arbitrarily fast network connection between you and the database. Uh so this is kind of cool, you know, you can give it the max time range and then I
[38:29] added a button here that will show all Fortune 500 companies all at once and then it does some down sampling, but I can turn that off. Um so this is a million rows, you know, just like rendered in 1 second. So uh incredibly fast um by using ADBC and by
[38:44] keeping the data in Apache Arrow columnar format like all the way through including to the canvas where it's rendering this chart, we just get blazing fast performance um with none of the typical bottlenecking that you get by sending this through, you know, JSON
[38:57] or um some other row-based format in the middle. All right. So just one more slide. Um so this is something I don't have time
[39:09] to fully demo, but one thing I want to talk about is how effective ADBC is for more of like full-fledged agentic analytics work workflows. So basically every analytics workflow, whether it's a
[39:21] human or an agent doing it, is kind of a loop between two things like reasoning and querying. So reasoning is like you're interpreting the result you got, you're deciding what to ask next, and then querying is like actually executing
[39:33] that request waiting for the result to come back. Um so agents dramatically compress the reasoning part. They can like inspect results, decide the next step, make tool calls, um keep moving faster, much much much faster than a
[39:46] human analyst. ADBC on the other hand compresses the querying part. It reduces the amount of time you spend like waiting on the connector to execute queries or return data. Uh historically uh with a human analyst
[39:59] and a slow connector um most of the elapsed time was reasoning time. You know, our human brains are are are not so fast. So, you know, I mean, we've all been there, right? If you if if you have
[40:11] a slow connector and you're doing some kind of analytics task, yeah, it's a little bit of a nuisance, it It friction, um but, you know, especially if you're doing like an iterative task like root cause analysis, but really it's not you
[40:23] know, usually the dominant cost waiting for for query times. But so if you you know, if you swap in a faster connector for a human driven analytics workflow, yeah, it shrinks the time a little bit, but
[40:36] not dramatically. Agents flip that equation. So once reasoning time shrinks down, the same slow connector becomes much more of a problem. So now the overall workflow
[40:48] spends like a much larger share of its time just waiting for the queries to finish. And this is where ADBC really matters because when you go from kind of like agent plus slow connector to agent plus fast connector, you're no longer
[41:01] just like trimming a small slice of the workflow. You're attacking like a huge remaining source of slowness. The overall time to compute the task drops by a large proportion. So I think this is really like the key difference
[41:14] from the the human case. With humans, faster connectors in Pre Vue X, they're nice, but with agents, they can dramatically reduce the end-to-end time. So I think yeah, this is kind of the slide to remember. Slow connectors were a nuisance, but now
[41:28] they you know, they're kind of becoming the biggest bottleneck. All right. I will share two links. One is if you have further questions
[41:40] or you'd like to chat with us, we have a community slack. And also we post a lot on LinkedIn, lots of new announcements about drivers and so forth. So love it if you'd follow us there. And
[41:53] I hope we have time for questions, but I think I went a little over, so I'm not sure. But yeah, thanks everyone and I'll be around if not here then down or out in the in the lobby to answer questions. And Matt and Emil will be
[42:06] around as well. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.