AI Data Is a Data Engineering Problem: Query Agent Logs on Iceberg in the Browser

Summary

Data engineering problem: AI logs need cheap storage, fast queries and LLM-powered curation.
Open formats: Parquet and Iceberg on S3 replace expensive SaaS tools with cost-effective storage.
Inverted indexes: time travel and log grepping cost 6% of data size with LLM-as-a-judge scoring.

AI Data Is a Data Engineering Problem: Query Agent Logs on Iceberg in the Browser

Talk by Kenny Daniel, CEO, Hyperparam
Storing and querying AI data is a data engineering problem rather than a modeling one: agent traces, chat logs and coding-assistant logs are text-heavy tables with one fat column, and they need cheap storage, fast queries and LLM-powered curation. This video builds an AI observability stack on open formats instead of an expensive SaaS tool, treating Parquet and Apache Iceberg files on S3 as the database, with open-source libraries including a Parquet parser, the Icebird Iceberg reader and Squirreling, an async agent-native SQL engine that runs in the browser.
Kenny Daniel of Hyperparam demos HyperStack, open-source libraries including a Parquet parser, the Icebird Iceberg reader and Squirreling, an async, agent-friendly SQL engine. See time travel over a 400 MB Iceberg table, grep over logs using an inverted index about 6% of the data size, and LLM-as-a-judge scoring, such as a sycophancy rating, run as a SQL function. Learn how to monitor and trace what AI agents produce, why traditional observability tools get expensive, and how a client-first stack also works for AI harnesses like Claude Code and Codex CLI.

Chapters

FAQs

Why is AI data different from regular data?

AI logs are text-heavy with one fat column containing entire traces, chat logs or coding sessions. They need cheap storage, fast queries and LLM-powered curation—not traditional schemas and indexes.

How do I query AI logs cheaply?

Use Parquet and Apache Iceberg on S3 as your database instead of expensive SaaS observability tools. Time travel shows data history, and inverted indexes let you grep logs at about 6% the data size.

What is Squirreling?

An async, agent-native SQL query engine that runs in the browser, letting you query Iceberg tables and AI logs without a backend server. It works with private data and auth systems like Claude Code.

Can I run AI scoring as a SQL function?

Yes—run LLM-as-a-judge scoring such as sycophancy ratings as SQL functions. This works for evaluating and filtering AI traces for quality and safety.

Full transcript

`00:00` Introduction

All right. Thanks everybody for coming out to my talk. I'm excited to be here. My name is Kenny. I'm the founder of a company called Hyperparam, and we're really focused on data engineering for AI. This talk was inspired originally by an argument I got into actually with an early employee of Databricks, my friend Denny Lee. We were out at a dinner, and I forget exactly how it came up, but he made the assertion that Databricks is an AI company.
And as you'll learn in this talk, I like to question assumptions, and so I responded a little impetuously, you know, in what way is Databricks an AI company? And a little bit of an argument ensued. And honestly, I agree with him on a lot of things. I do think that Databricks is deeply involved in AI in so many different ways. And upon reflection, I think what actually offended me or annoyed me about the statement that Databricks is an AI company, that it felt like it was forsaking their roots as the original data engineering company.
And I love data engineering. I'm a huge data engineering nerd. I love Databricks. And that's why I understand everybody's going to be chasing AI and wants to put themselves in that bucket. But there's so many interesting questions when it comes to data engineering even in this new world of AI So I don't think it's going to require you know much evidence here to back up But you know Google is on the record saying they're producing over 2 3.

`01:26` Why AI logs and token volumes are exploding

2 quadrillion Tokens per month and it increased 7x over the last year and no signs of slowing down Over 30% of enterprises pulled by Deloitte Reported that they were generating over 10 billions of tokens per month And so in this world where we're being faced with a wall of tokens that are coming at us, there's a question of what do you do with that data? And importantly, I don't think this is controversial either, there's a huge amount of value in that data that a lot of people are, broadly speaking, not looking at it.
Now, I'm talking about multiple things here. I'm talking about production AI logs that are being used in production systems with customer chatbots and all that. But I'm also talking about things like your company probably has hundreds of developers that are using cloud code. Are you even collecting the logs for that? Probably not. And if you are, are you getting insight out of that? Are you able to query it? Are you able to feed that back into your business so that you can actually benefit from all of these tokens that you're spending on?
And my observation is that a lot of companies are not actually getting the full value that they could out of their AI data. So to get a little bit more into the weeds, like I said, this is a data engineering talk. And so this is the way that I look at the world. This is a representation on the left showing sort of classic BI, OLAP-style data, where you have a bunch of columns. They're mostly structured, numeric, categorical, click data, event data, things like that.

`02:34` AI data looks different: one fat text column

But we're very quickly moving to this world on the right, where the vast majority of data sets that we're going to see for the rest of our lives have a couple small columns of numeric, categorical, IDs, things like that. And there's going to be one fat text column. In a lot of cases, there's going to be conversation logs, agent logs, you know, customer chat logs, all these sorts of things. And how do you actually do data engineering on this, right?
This probably, this is not the classic thing that database and data engineering systems were built to work with. So I think there's an opportunity for rethinking this from the ground up. Obviously, I'm not the first person to think about what does AI data look like, how are we working with it. There's all these companies that are trying to get into this space. One thing you may notice, though, is a lot of these are companies that existed before the advent of AI and before ChatGPT.

`03:40` Why existing AI observability tools are expensive

And these are more classic APM systems, or more classic tracing and OLAP-type systems. And sure, everybody's trying to adopt AI, so they're coming up with AI solutions. But they kind of weren't built for this world. In general, if you look at them, they're expensive. You pay for ingest in a lot of cases. You pay for storage. And then they also charge you for the privilege of querying your own data. And this is also complex.
This is adding more complexity to your stack. There's a third party dependency, and you're dependent on them not raising their prices in the future and things like that. And also importantly, in a lot of these cases, you don't own your data. You're sending your data over the wall to a third party, and you're trusting that they're going to keep it secure and private, and that if you ever want to migrate in the future, that it won't be a huge pain.
And so I wonder if in the edge of AI, if there's not a better way that we could do this by thinking sort of from the ground up, what would the system look like? So a little bit of an aside, there's this new class of systems, which I really like, which are cloud native systems. Iceberg is one of the canonical examples of this, but there's been new companies that are coming out that leverage object storage, S3, as the world's most scalable and cheapest database, and building on top of that.

`04:45` Cloud-native systems: S3 and Iceberg as the database

So that's things like Iceberg, ClickHouse is doing really well in this space, TurboPuffer is building a vector database on top of S3, and we're introducing a set of libraries called HypeStack, which is aiming to do the same thing, but for AI data. And if you can make these cloud-native systems where S3 and object storage does most of the heavy lifting, you can get a huge amount of benefits. Now, there are trade-offs.
You need to work really hard with the engineering and the architecture in order to make this work. But if you can do so, it's really compelling. But even in this cloud-native world that we live in, maybe you're using Databricks, maybe you're using any of these other systems, if you wanted to build a tool for users to interact with their data and get insights out of their AI data, you would probably build a system that looks something like this.

`05:50` Deleting the backend: a browser-only data stack

This is a very classic sort of application architecture. If you go to ChatGPT and you ask it to build a system for viewing and querying data, it'll probably come up with an architecture diagram that looks somewhat like this, if not substantially more complicated. And it's got everything from the browser, talking to CDNs, going through load balancers to the front end stack, which scales independently and makes requests to the back end stack.
And then it sends requests to a query engine to make requests against iceberg or wherever your data is living. And this is fine. This is great. But as engineers, I think we should aspire for simplicity as much as possible. And in the spirit of the best part is no part, What of this could we delete if we just wanted to take a marker and delete as much as we could from the stack? Well, you can't delete S3 and object storage.
That's where your data lives. And you can't delete the browser, which is where the user lives. But everything else is maybe fair game for deletion. And if you could do this, you would end up with a stack that looks like this, which is almost a laughably simple stack. But if you could do this, it would be very compelling. So this is what we set out to do. We wanted to remove as much of the back end as physically possible and move everything into the client that we could.
One thing of note is that the browser and a lot of these client-focused applications, they're mostly in JavaScript. You need to be able to run the things there. There weren't a lot of data engineering tools in the browser and in JavaScript when we started this. But we needed ways to, first of all, store the data. We needed ways to query the data and the queries of this particular type of data, which we'll come back to.
And you need to be able to integrate this into your systems of auth and things like that. So to do this, we built a stack of open source libraries. We'll go through a couple of these through the course of the talk. But these are all open source, dependency-free implementations of Parquet, Iceberg, Query Engines, Grep, Vector Stores, things like that. And we built this from the ground up, some of them even before AI, some of them with the assistance of AI more recently.

`07:44` HyperStack open-source Parquet and Iceberg libraries

So going into it, if you wanted to build this stack for AI data, first you need the storage layer. And for this, we're not reinventing the world. We want to use Iceberg. We want to use this cloud-native thing of using S3 as the world's most scalable database. And Iceberg has all of these nice properties, some of which it just inherits from Parquet, which is column-oriented. It's range-readable, meaning you only have to fetch the data that you need if you only want a subset of it.
There's a wide variety of compression options and codings that you can leverage to efficiently store this data. And it supports push down statistics and queries so that you can avoid fetching a huge amount of data. All these properties, which made it great in the warehouse, actually work really well in the browser, where you really want to optimize for minimizing network transfer. That's the biggest bottleneck there.
And then you can, so we built from scratch a Parquet parser, which is now the most popular Parquet parsing library in the browser world. Once we had a Parquet parser, we built an Iceberg parser that we call Icebird. And by doing this, we can do some magic tricks that would not really very easily possible in the world before this. So for this demo, the rules are there's some data stored in S3. And this entire page here is live.
And it's hosted on a GitHub page, like a GitHub static posting page. So there is no server whatsoever, except if you count S3. And what this can do is it can load the data really quickly, write it in a browser. Again, very, very simple stack. You can even do things like time travel. So you can see 1,200 rows going up over time. So that's pretty cool. But looking at your data is just one thing. But really what we want to be able to do is we want to be able to query over this data.
And when we talk about querying AI data, there's a couple things that you want to do. One, we need SQL. SQL is table stakes, so to speak. Because there is structured data, we want to be able to filter by user or by session ID or by things like that. But we also want to be able to do these other things that are more useful against text, things like grep and things like applying LLMs across the data. So we'll come back to that.
So our whole shtick is kind of reinventing things from scratch and building it from the ground up. And so rather than use an existing query engine, we built our own. We called it squirreling, following the long tradition of using words that have SQL embedded in them. And we made a couple interesting design choices in squirreling that didn't exist in any other database engines. The biggest one is being asynced to the core.

`10:18` Squirreling: an agent-native SQL query engine

So every individual cell is asynchronous and lazily computed. It's not sort of a cursor kind of output structure. And there is a very good reason for this, is that the data, first of all, in these column-oriented stores often come in. The small columns come in very quickly. The large text columns take a little bit longer. And we'll also talk about wanting to do very expensive UDFs, things like running LLMs across it.
But there's one other interesting choice, which I kind of wanted to talk about, which is by building this from the ground up, we were able to build a query engine that is built for agents. And we actually just published a paper in the ACM conference on AI and agentic systems. I was very proud. It actually won best paper in the Supporting Our Agent Overlords workshop. And in it, what we did was we studied squirreling our agent-native query engine versus other ones like DuckDB and Postgres. and one of the things that we did is, unlike a system like Postgres, we are multi-dialect.
So almost any SQL syntax that you can think of that is not directly in conflict with other ones, we try to support it. So we have aliases for all sorts of different functions, for aggregates, for all of these sorts of things. And what's nice about this is that when an agent tries to make a query, if you can get fewer rounds of iteration, you can get an answer faster and you can save money and you can save tokens. and so but we don't it's you know if you're making postgres and you support a particular syntax you probably need to support that for the rest of time because somebody's going to come to depend on it but with an agent i don't really need to worry about that as much like the agent will just keep trying and we can evolve the engine pretty quickly and do that the other thing that we worked really hard on with this to make it agent friendly is we worked really hard on the error messages because if you want a model to know something you need to either put it in the context or put it in a tool description, or you can give it to it on demand when it hits an error.
By doing that, if it requests a function that doesn't exist or it uses a syntax that doesn't exist, if the engine is able to say, hey, maybe try this function or maybe try this approach, the model can self-correct. So this is kind of different than the use case that most databases have been built for. And we were able to show statistically that you can save money and get faster answers if you do this. So we'll give a quick demo.
And this is all live. This is not a video. And again, this is a statically hosted web page pointed to S3 with no servers in between. And this is showing an aggregate by day. And again, I can do the time travel, which is kind of cool. And you can also see it's really, really fast. And this is a over 400 megabyte iceberg table. So it's not a trivially small thing. But you can see, over time, more days accumulate into the system.
And you can imagine doing more complicated queries on this kind of thing. So OK, SQL is cool, but it only goes so far when we're talking about large text data sets. It's really great for aggregates, but if most of your data is text, you probably want something that is more oriented to text. And if you watch Claude for even five minutes, Claude code doing its thing for five minutes, you'll see that agents love to use grep.

`13:22` How to grep AI logs on S3 with a Parquet inverted index

And so a natural question was going to be, OK, how can we grep over these AI logs that we are now accumulating in S3? And the classic answer to this would be something like Elasticsearch. But the problem with Elasticsearch is you're going to have to keep a bunch of server infrastructure running. So you're paying, even if it's in just, you know, cold state of not being queried, you usually have to use pretty beefy memory machines.
And so we started asking the question of, could we build a cloud-native approach to grep so that I can shove all my AI logs up in S3 but still be able to grep it? And so we built, using another parquet file, we built an index, an inverted index into a parquet file. So if you have, in this example, a 400 megabyte Parquet file or an iceberg table, you can create an index that's about 20 megabytes, about 6% of the size of your data.
And we do some clever Parquet engineering under the hood using things like the Delta byte array encoding and the Delta binary packed encoding and some cleverness of how we use the row groups and the column chunks and the column indexes. And in doing so, we were able to build a grep system on top of your data, which when it's idle, it's practically free because it's just S3 cold storage. But when you want to go query it, you can do it.
And again, this is all real time, not cache, not whatever. And what it's doing under the hood, it's going out, it's reading that index, it's looking for the ngrams that are in, you know, in this case, certainly. Models love to say certainly. And finding, okay, here is the map of rows that I want to do. and then it's using the parquet range read index to fetch just the data that it needs. So now, great, now agents have the power of grep.

`15:16` Run LLM-as-a-judge as a SQL function

But the kind of questions that we want to ask of this AI data can't always be answered with grep. That's a keyword sort of based search. But a lot of these things really, we need to understand the text. Which model responses fails? Where did the model go down a rabbit hole? Where did our users leave the chat frustrated? it? And also importantly, how can I use this feedback into my systems? No regex is actually going to help you to answer these questions.
You need understanding of the text. And that means either a human needs to read it, but nobody has patience or time for that, or we need to figure out as an industry how we can integrate LLMs with our query engines. And we need a way that we can run these LLMs across the data in order to understand it and extract what we're trying to understand. There's a lot of ways we could do this. So I'm not going to say that this is necessarily the best way or the only way, but one of the things that we've explored is we really wanted to be able to run LLMs as a user-defined function.
And this goes back to what I was saying about squirreling. It is async to the core, so it will run async UDFs in parallel when they come in, even out of order. That's fine because everything is async. So in this next demo, what you'll see is a SQL query against some data, and you can kind of see there's a sycophancy score, so it's a rate on a scale of 1 to 10 how sycophantic this model response is. Everybody hates sycophancy.
And what you can see, I'll do a refresh of the page here, hopefully it won't break, and you can kind of see that it doesn't fill in one row after the other. It happens in parallel, and when the response comes back in, you can see the cell fill in. And you can use this for filters and for sorting and things like that, which is pretty cool to be able to do directly in the browser. And I think this is one of the ways that you can integrate LLMs into your query engines in order to understand these predominant data sets that we're going to be seeing.
Cool. So we've got the ability to store it efficiently using Iceberg and open formats. We've got query engines that can do structured queries. It can do grep queries. It can integrate LLMs. But I know what you're thinking. Cool. but my data is not in a public S3 bucket. I can't just point to a GitHub page at my private company data. And there's actually solutions for this. I will actually say on the previous demo, I actually logged in via OAuth using AWS Cognito because I wanted to be able to call the LLM, but the same mechanism of authentication entirely in the browser without a server you can also use to access your private corporate data.

`17:21` Private data, auth and AI harnesses like Claude Code

But there's another interesting use case here for moving the entire data engineering stack to the client. And I know I've talked the whole time kind of about the browser, but there's this new class of application, which I think is increasingly important in our society, which is AI harnesses. And these are things like Cloud Desktop, Codex CLI, Cursor. And in some cases, these are CLI applications, but they're architected a lot like front-end applications.
Cloud Code, famously, is actually a React app that just renders to the terminal. You know, Cloud Desktop is built on JavaScript because it's an Electron app. Cursor, also VS Code, all JavaScript. And so another interesting part about these applications is these are not architected in a traditional way. There is no server component of these applications. I mean, there's the model. Sure, it's calling AI. But like Cloud and OpenAI are not hosting a Python server somewhere that is supporting Cloud Code.
It just runs locally. And so it has these exact same problems that if you wanted to access your data from one of these agent harnesses, there's a question of how you would do it. Would you have to create some backend infrastructure that it calls? Would you have to bundle some Rust or some Python, which is hard to run in client systems? Or you could use something like we built with HypeStack to move the compute processing as much as possible into the client and then just point it straight at storage.
And if you're in one of these AI harnesses, the auth problem kind of goes out the window. They can just inherit directly from you as the user. They can use your local AWS STS tokens and things like that. This is part of the reason why Cloud Code and Codex CLI were so popular is because they were able to run in the user's environment and therefore able to get access to this data that they otherwise wouldn't have been able to get access to.
And I think we can take that farther using this kind of client-first approach to data engineering. So, okay, these were some cool demos. What does this actually give you as benefit? Why go through these hoops? Well, for one thing, it can be drastically cheaper. And so there's a worked example on these numbers. You can check my math if you want. But broadly speaking, if you go to any of these sort of providers of LLM and AI observability, it's going to be quite expensive.

`19:51` Cost and simplicity benefits of a client-first stack

You're storing it in some cases on their server, you're paying for compute, you're paying for storage. If you host it yourself, this is the sort of iceberg insight, you can do this much, much more cheaply. From a developer point of view, it's a much simpler architecture. This is easier to build. There's less things to be worried about. There are less systems to be paged about in the middle of the night if you don't have as many boxes on your architecture diagram.
And the points I was making of, if you do this right and build this from the ground up, there's advantages you can get for this world of AI. You can build it for agent consumption from the ground up. So bringing it back around to our original point, you know, Databricks is an AI company in the sense that if you're anywhere other than OpenAI or Anthropic, your experience of AI, the problems that you face, are probably much more like data engineering problems than they are like modeling problems.
And the questions that you're going to be asking yourself is how do I store this data? How do I query this data? How do I learn from it? And how do I feed this back into my company knowledge base, my stack of markdown files, my prompts, my skills? And if you can understand your AI data and get this insight of it, that I think is how we build better AI systems and build better companies in the age of AI. So, happy to, thank you.
Happy to chat with people after.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.