Query Expert: Building RAG-Powered Data Agents at Enterprise Scale

Summary

Block built Query Expert, an AI data agent used by over 3,500 employees to self-serve analytics across the company's five business units and 20,000 tables, launching in July 2025.
The system uses a three-vector-database architecture with Databricks Vector Search, semantic reranking, and human-in-the-loop verification workflows to return answers with confidence scores.
Automating table documentation and mining institutional knowledge from query history codifies expertise that previously lived in Slack threads and individuals' heads, reducing the ad hoc burden on data scientists.

Query Expert: Building RAG-Powered Data Agents at Enterprise Scale

Building institutional knowledge into AI agents requires more than RAG. Learn how Block built Query Expert MCP, a production AI agent used by over 3,500 employees to self-serve analytics at enterprise scale. Discover how to automate documentation across 20,000 tables, mine institutional knowledge from query history, and orchestrate retrieval-augmented generation with Databricks Vector Search and Pipelines. This talk covers the three-vector-database architecture, advanced semantic reranking techniques, human-in-the-loop verification workflows, and real-world patterns for maintaining context accuracy and freshness in large organizations.
You'll learn the complete workflow: how Query Expert finds the right data sources, discovers domain experts, validates queries against business rules, and returns actionable insights with confidence scores. Whether you're building internal data tools, multi-tenant AI platforms, or autonomous analysis workflows, these enterprise RAG patterns will show you how to scale context engineering with Databricks.
🤝

Chapters

FAQs

What is Query Expert and who uses it at Block?

Query Expert is an internal AI data agent built by Block that enables employees to self-serve analytics across the company's five business units and 20,000 tables. Over 3,500 employees use it as of its July 2025 launch to get data-driven answers without relying on data scientists for ad hoc requests.

How does Query Expert's three-vector-database architecture work?

Query Expert uses three separate vector databases to index table documentation, institutional knowledge mined from query history, and business context in a dedicated knowledge store. This layered approach allows the agent to find the right data sources, discover domain experts, and validate queries against business rules before returning results with confidence scores.

How does Query Expert automate documentation for 20,000 tables?

The system uses automated pipelines to generate and maintain documentation across Block's 20,000 tables, removing the manual burden of keeping table descriptions current at enterprise scale. This automated documentation feeds the retrieval layer so the agent can surface the most relevant data assets for any incoming query.

What retrieval techniques does Query Expert use to improve accuracy?

Query Expert applies advanced semantic reranking and custom reranking techniques on top of retrieval-augmented generation to improve result relevance. Human-in-the-loop verification workflows allow domain experts to validate outputs and correct the system, maintaining context accuracy and freshness over time.

Full transcript

[00:09] Hey everyone, uh thank you for coming to our talk on query expert, how we built an internal data agent at block that's used by over half the company to self-s serve insights. In this talk, we'll talk about why we built query expert and how we built it with data bricks. Plan to talk for about 30 and 35 minutes to leave some time for
[00:26] uh demos and questions at the end. My name is Phil. I'm a machine learning engineer on applied AI at Block. I've been at Block for four and a half years in various data roles. And for the past 16 months, I've been working on internal AI tools for data scientists at Block.
[00:44] And my name is Dan. I'm a product data scientist at Block. Um, I primarily still do product data science as my day job, but I was involved in a lot of the early prototyping and building out some of the more narrowly scoped versions of the tools that we're about to present today and have been an early uh super user and alpha tester along for the ride
[00:59] the whole way. So, I'm going to set the stage for the problem we're trying to solve. Uh, to do this, we're going to go back in history to 2025. Um at the enterprise the scale of block data knowledge was spread across tools
[01:15] and tables and how to work with it you know lived with various do various Google docs and slack threads in people's heads finding the right data or tool to solve your problem was difficult and timeconuming block is a tech company with five business units and just to name a few we
[01:31] have merchant processing data with square we have peer-to-peer and banking data with cash app and we have artist catalog and streaming data and title And within those businesses, our whole ecosystem of risk, marketing, product, and operation teams with different data
[01:47] needs and tools. Each team manages their own data solutions and data practices vary with each team. Without centralization or single source of truth, stakeholders and data teams are confused on where to go and where to find data. As a result, data scientists are the only uh ways we
[02:04] can work with data and they become burdened with ad hoc requests. um they become burdened with ad hoc request and become barriers to become a data driven or work rather than facilitators to become a data driven or
[02:26] is that not working? Yeah. Um as a result blog data team slide channel looks something like this and all these questions are kind of asking the same question. Who is the expert? Who can I talk to about this data? Who can validate my assumptions? and who can I work with? The title here is the exact insight that led us to build query
[02:43] expert. Dan and I were on the uh Square uh payroll team in early 2025, the uh tax reporting PM uh which is a feature suite that has Square sellers uh reports their W2 and I9's for their employees. Wanted to build a more cohesive
[02:59] experience with Cash App tax filing, a feature available to 60 million Cash App customers that allows them to file their taxes in the app. In order to understand the cohesion between Square Tax reporting and cash app tax filing, we need to understand the funnel, the potential opportunities, the customer profiles, all the things that data
[03:16] scientists do. But it took us two weeks just to find the right data team, the correct table, the query. And before we can even begin an analysis to inform any decisions, the data team became the bottleneck to product and engineering uh
[03:32] to move fast during a busy tax season. It was this finding that led us to the architecture decisions that people need to find the expert and know who to validate their assumptions against their work.
[03:48] Now enter AI. Now we're in like March 2025 at block and teams are experimenting with AI tools to self-s serve data. But this all varies by team. We now had an AI tool strategy that resembled our internal data. It was spread out, hard to find, and completely incompatible with each other. So each
[04:04] team constarted creating like their own bespoke tools. None of them worked with each other and people didn't know where to go to find accurate AI assisted insights. And we saw more than ever a lot of vibe coded SQL with madeup table names,
[04:19] column values that didn't exist, and people sharing unverified insights. So we needed to have a centralized solution with governance and a tool that allowed teams to work together instead of on top of each other. So our goals became clear.
[04:36] We need to build a data science co-orker that thinks and acts like a data scientist. To achieve this, we need to first invest into a centralized data and context stores. When you ask a data scientist a question, they implicitly have the knowledge of what are the correct tables
[04:51] to use? How are those tables produced and the assumptions baked into the data? What is the business background and context of this question? and what are some common metric definitions. Then the data scientist writes the query, validates the queries adhering to metric
[05:08] or business or platform rules, executes the query and then produces actionable insights. This is the workflow we want to codify into our data agent. But all depends on first being able to retrieve and understand those top level concepts and
[05:26] how they relate to each other. All right, at this point we're going to show a quick demo just with slides um of what a basic data discovery or onboarding exercise would look like using query expert. Then we're going to
[05:42] circle back and go through in detail how query expert knows all the things that it knows in the reply that it's going to give here. Our prompt is going to be fairly simple. Use query expert to find who I could talk to about cash app taxes data. Also tell me some typical query patterns and key metrics in that space. I'll demo
[05:58] this in our internal agent tool, goose, which also has an open source version. If anyone's interested, you can find it. Um, you'll see here that the first thing um goose does is it's going to reference the query expert skill, which is a bundle of instructions that tell it how
[06:14] to use the tools that the query expert MCP provides. You'll then see that it's going to do a couple different tool calls under the hood to retrieve information from three different search indices that are being served by data bricks endpoints. We'll circle back in a minute to talk about how we built each of these endpoints and
[06:29] how they're served. At this point, the agent thinks it's found the data and information that it needs to answer my original question, but if it needed to, it could continue to iterate further by continuing to query against these search indices that it has access to using previous results to inform the iterative future searches.
[06:47] This is just more of the reply that the agent gives in our harness here. Um, you can see this is going to tell me a handful of people and their different areas of expertise within this data domain. You can see it'll also tell me key tables in the space and even a few key fields within some of these tables and how they might be used to define metrics
[07:03] or objects of interest. Um, and then finally, it'll have actually some kind of like business logic input at the top here commenting on how a funnel might be defined in this space, which is great because this would be a new area for me that I don't not going to know the product funnel off the
[07:18] top of my head. Um, and then finally, it'll list even some specific query patterns um, from those experts it found and give me some common gotchas and things to be aware of about this data. I'll specifically note here that it's flagging um, that this table is unverified at the very bottom of the response here. We'll circle back to what
[07:34] that means in a minute as well. So, at this point, I might be ready to go follow up with these actual human beings um, and ask them questions if I wanted to, but really in 2026, the goal here is that my follow-up questions would also be against the same agent. Um, so in that with that mindset, you can think of query expert as a context engineering
[07:50] tool. Um, we just loaded up our agent with a bunch of specific bespoke context to this one specific area that includes both business and data documentation. Um, and I'm now in a really great spot to ask follow-up questions and dive into any analysis that I want to in this domain.
[08:11] Go back, Phil. Cool. Uh so now we're going to get into the weeds of how we built this agent uh that's backed by datab bricks. Data bicks really helped us unlock the scale for us and enabled a rag architecture that is resilient, accurate and flexible. So first I'll take a step back and
[08:27] quickly explain rag and why we invest in this architecture others. Uh, retrieval augmented generation enables natural language search across your company's entire knowledge base by embedding the user's question and semantically matching that to relevant information.
[08:43] An agent can ask any natural language question. That question is embedded um and then match that uh knowledge layers and then it's retrieved. The query expert MCP is a Pythonbased package that establishes the cool tool contracts with the agent. So it knows how to search and
[08:59] filter for relevant knowledge. And the MCP also contains custom logic, SQL building and validation before returning the results back to the agent for understanding. So we chose this architecture because of the scale of our knowledge base and the need for an agent
[09:15] to be able to search, traverse, filter that knowledge. Rag enables that enterprise scale without sacrificing that domain specificity that we need. Getting all this contextual information pipeline and embedded and served in a low latency way is where data bricks
[09:32] vector search and pipelines play an important role. Query experts underlying infrastructure are three vector databases that mimic the knowledge a data scientist uses to answer any data questions. With data bricks pipelines, we keep this
[09:48] information up to date and timely. Each pipeline is ran on data bricks using data bricks sub aents to help label and enrich raw information then chunks and embeds all blocks data knowledge into semantically searchable index that's retrievable via the data bricks endpoints.
[10:04] We will dive deeper into each pipeline and how they're created but at a high level uh we have tables. We have the semantic model of all highly used tables at block that meet certain guardrails of quality so the agent can find the right table and columns for their use case. We have queries. We have metric and high
[10:20] informationational queries enriched with metadata that inform the agent of the deep nuance of authoring the correct SQL syntax including join logic and filter validations and business knowledge. This is a kind of orthogonal uh database that houses you know wide reaching business strategy
[10:37] and deep domain knowledge and context for the agent to better reason and understand and deliver insights. It's important to call out that those three uh pipelines I showed are not uh
[10:53] um exclusive swimming lanes. They're not silos. Each vector database contains primary keys and foreign keys to join with each other. This is crucial for allowing the agent to search, iterate, and remain domain specific. Data bricks vector search APIs allow for filtering of the search. By architecting our
[11:09] schema so the outputs of one search can be inputs into the another, we create a way for the agent to hone in on specific domain context as well as find new information that might be relevant to the user's question. So query expert can find the table, find the queries that
[11:25] reference that table, discover relevant knowledge knowledge docs for that domain and so forth. This also allows users to kind of set uh a domain on top of when they're working with query expert uh that allows kind of
[11:40] a single architecture that can serve those five business units and 40 plus data teams we have at block. So when a finite financial analyst asks a question query expert knows it's the financial domain use financial tables and find the financial queries and metrics the same way a marketing analyst might ask a
[11:57] similar question. All right. Next, we're going to talk about how we built our table documentation index and how we scaled this process to document all of our tables or a lot of our tables. So, the sheer scale of data
[12:14] block gives us both a cold start and longtail problem for documenting our tables. Um, so if we were to truly attempt to document every table or object that exists in all of our databases, the number is something like 200 million. Um, this is an impractical scale and also most of these are not relevant to analytics anyway. So with
[12:31] some basic filtering around which tables are actually used and used recently, we can whittle this down to something in the tens of thousands. But that's still a big number and requires a scalable and automated solution for documentation. This is where we leverage multiple pieces of data bricks infrastructure in combination with our own query history
[12:46] and lineage data to create semantic models and description of each of these 20,000 or so tables. So getting into the details, one of the main functionalities we leverage is data bricks AI query which this is just showing the data bricks documentation here. Um allows us to run LLM chat
[13:04] completions inside SQL itself. Very scalable. So to see what this actually looks like, um here is the uh templated part of the ETL that enriches our table descriptions with uh data based on how this table is
[13:20] actually used by expert queries. So you can see we assembled this prompt that's going to feed an LLM some metadata about the table, deterministically available column names and data types, and then also the example queries from our internally logged query history that were run against this table by data
[13:36] scientists and that's filtered down. significantly. We're then going to ask the LLM to provide a rich description of this table, how it's used, and the analytical questions that it can help answer. This is what the table description output looks like for a nice successful run. The idea here is to optimize for
[13:53] the rag searchability. We want to pack these tokens full of as much uh specific semantically aligned language as possible with what data actually lives in this table and the sort of analytical questions that it could be helpful to answer.
[14:09] The same concept of feeding example queries to an LLM to help provide automated documentation applies to columns here as well. You can see we get some nice rich column descriptions around how columns are actually used in queries, what they help measure, and what they might be joined to.
[14:27] As a part of our automated table documentation, we also create this bundle of metadata associated with each table, which includes a variety of other fields that'll be provided back to the agent. These help give additional context to this table and also help the agent traverse our knowledge and documentation graphs by giving it useful related entities like people in tables
[14:44] that could be uh used as subsequent search terms as the agent iterates to find the right knowledge. Um also call out here that you can see we have a verification status in this metadata. In this case the ter table is verified which means that a human being has approved of the autogenerated
[15:00] documentation or has made edits to it themselves and then approved of it. This verification status is provided back to the agent so that the documentation can be considered with appropriate weight and confidence. And verified tables are also boosted in our rag search scoring. So all else equal, the agent is more likely to find a verified table than an
[15:16] unverified table. Lastly, this just shows a quick snapshot of what actually running this pipeline looks like on a day-to-day. Um, it's a data bricks ETL, runs in about an hour and gives us a summary of what it did on Slack. So you can see uh in this case it updated 237 different table documents.
[15:34] Some combination of adding new tables, changing columns in existing tables and deprecating tables that become stale or unused. Cool. All right. Now we're going to talk about that other index, the queries. And this
[15:51] is a very important index in query expert. So, your query history is a gold mine of institutional data and business knowledge with rich metadata that you can extract. This source is extremely useful for understanding the most up-to-date and relevant information about how your company uses data without
[16:08] taxing teams and IC's to manually contribute. However, mining it is difficult. We have millions of queries we run each month at block. So, how do we know which ones are useful or correct? So first we filtered to only queries that are ran through any any of our BI tools or written by a data
[16:24] professional. That's our first guard rail on accuracy and reliability. In our final corpus about 75% of queries are from BI tools and 25% have been uploaded by data teams. Next we queries by the information in its text. So we score the query uh
[16:41] biasing it to capture more join patterns queries with aggregations and unique filtering. So we now sample these queries down to represent representation of tables making sure that we have comprehensive coverage of all the tables at block in our query index. We then sample again by users to make
[16:58] sure every data professional at block has an upto-date data corpus of their queries. Finally, we enrich the query with a labeled description, information about how the user in the data domain and embed all that information back into our
[17:14] data bricks vector search database. We wanted to allow a low friction way for data scientists to add a nuanced context to their queries and keep this corpus up to date. So, we met them where they were when they're running an
[17:30] analysis or creating a metric. So here uh when a data scientist authors a query they or more likely their agent uh can apply the query expert label which is just a comment in line with the SQL. Uh so this places the context right
[17:46] where the data teams are already working and it's captured by our pipelines for added enrichment. Label queries are very important queries in our search index. We parse that description and embed it to be reranked in a semantic retrieval. We have customer ranking logic that
[18:01] weighs these queries higher to be retrieved by the agent. And these queries are also the primary seeds of information that are used to create and update those table and column rules in our table corpus.
[18:17] So just to bring it alive a little bit, um here's an example of what the final few rows look like just before the embedding process. Um and what what we do with all the information waiting. So we're indexing on who ran the query, you know, who is the query expert. Um we're balancing that corpus by the tables joined in the query and that becomes the
[18:33] foreign key to the tables vector database. This also contains join patterns which are broken down to a later serving step which I'll show. Um we also have the query description. So my last slide uh this would have been the comment to query expert label. But here this query is from looker which is
[18:50] a verified BI source. So our query description contains what the dashboard it came from and what the metric this query is measuring. A BI's query descriptions are fully automated within the pipeline. There's no tax on teams to contribute or to manually keep this context fresh. This query description
[19:07] metadata facilitates both semantic and custom reranking and adds that rich context back to the agent. So when a user asks about something that's related to this metric name, uh, query expert knows that it's from this metric. It comes from this dashboard and here's the query that produces it.
[19:25] We're going to kind of take a little bit of a side quest and kind of bring it to life a little bit more. Are we on my screen? Very cool. Um I didn't want to upset the demo god so I
[19:41] pre-anned all this but uh so um so essentially we're here in our internal version of goose. We have an open source version of goose that is available um through the AF foundation. Uh but this is our internal agent
[19:57] harness. And so I'm going to kind of pull out some of those like subprocesses to understand how they work. So first um I'm going to ask it you know how do I do this like complicated join pattern. So how do I join our dim user table with
[20:13] segmentations to get both historical and current. So I asked this query expert and in about less than 30 seconds it comes back with exact patterns about how to join these
[20:29] two tables and how they differ for both historical and current. This is like a so again this is all embedded in the graph database for how this table is how these two tables are commonly joined.
[20:51] Next I'm going to break down to um query validation that validation stuff that we get again because we have all the upto-date corpus of queries query expert knows the most up-to-date business knowledge around the query. So here I'm throwing a junk query at this
[21:08] uh agent. Um I actually made up these column names. So is time card. Here I'm asking about how many number of employees are active on time cards. That's a square uh POSOS feature where employees log into the square POS and it starts tracking their
[21:24] time. But here in this query I actually made up these column names. You know where is time card equals true and time card create day stamps and again see if like query expert can find it. So in less than 20 seconds, it actually
[21:41] knows to say like this query is not valid for the number of employees active on time cards and actually comes back to me with the correct query. Again, it knows this for free in its brain simply by mining that query history
[22:06] Next, I'm going to talk about the third component of our three index system. Um, which we call the knowledge store. So, the knowledge store is just the best name we could come up with for this or third orthogonal direction of information um that we think is useful to provide to the agent. So, as we
[22:22] developed this MCP, we used it in practice and iterated with it. We kind of observed that we needed this other layer because just the table aligned documentation and the query aligned documentation wasn't quite enough to stitch everything together and make this feel like a fully capable data analysis partner. The knowledge store layer is designed to
[22:38] complement the other two search indices and give our agents critical business product or customer context that will help inform how we analyze data in our ecosystem. This includes things like demographics, benchmarks, product and business strategy definitions, terminology, and lingo. We typically uh
[22:56] sorry uh the knowledge store also acts as connective tissue compounding the value of the other search indices by providing the agent with higher level concepts and relationships between entities that it'll find in the other indices.
[23:12] The knowledge store implementation generally mirrors the other two indices. We use GitHub as a source of truth in this case to store a curated set of documentation, leveraging GitHub's polished surface areas for contribution, review, and automation. From there, we load the GitHub documentation into a data bricks ETL and create a searchable
[23:28] index just like we do for the tables and labeled queries. The agent can then use a tool call to search against this index as well. Cool. All right. So throughout um throughout
[23:43] this uh talk we've we've mentioned words like semantic and custom reranking and all that stuff. Um so here I'm going to kind of deep dive these more like advanced and optimized rag techniques that you could probably take home with you. So all semantic retrieval steps in query expert follow this left to right
[23:59] flow. So we overfetch requests filter and rerank them and only return the most relevant information uh back to the agent using a score boundary and progressive disclosure. So filtering like I mentioned like our scheas are set up so the agent can filter and search
[24:15] down. Um and the MCP layer has the full knowledge and tool contract to allow those filters to be dynamic. Then we have semantic reranking. So this is a data bricks vector search capability that does a hybrid retrieval on those metadata columns or indexes. So
[24:32] for example that query description I talked about earlier um or or the uh column names out those two steps we get the semantic score from the data bricks vector search on a scale of 0ero to one about how relevant that information is to the
[24:47] user's original question then we perform an adjustment score based on our own internal logic. This is a custom reranking that can ensure the context you want and validations are met to surface at the highest relevant context comes back to the agent.
[25:03] Then finally, we control what the agent finally sees with a score boundary and progressive disclosure, which I'll touch on in the next few slides. So, I'm going to deep dive into that custom ranking knowledge and enrichment that happens before any context is sent
[25:18] back to the agent. Our custom reranking logic builds out a normalized adjusted relevant score based on that original semantic score from search. So we want to ensure the information is correct. So we focus on those metric and labeled queries. Those get weighed higher.
[25:35] Business knowledge can become out ofd. So we want to penalize older information and boost newer information by uh measuring a time decay penalty. uh so you know a query ran one month ago gets supersedes a query that was ran 12
[25:50] months ago. We also validate any join logic by using the probability distribution of that logic between the two tables. Um kind of like in that demo I showed where I just asked how do I join these tables. It showed that majority of the time this is how you join it.
[26:08] Finally we want to bias our agent to use that gold layer tables. So we, as Dan mentioned, we rerank to use verified tables. Um, so that makes sure we're using those golden layer tables. On the right hand side here, I have the example payload uh that comes back to
[26:23] the agent. So we have that adjusted score, which is uh an adjustment of that original score. We have that query description. There's a labeled query uh that I uploaded. We also have some metadata about that query. And then we break out those source tables and the join tables to try to really spoon feed
[26:41] the agent about how to work with this data under the context that asks for. So I'm going to talk about that progressive disclosure step real quick. Um one lesson we learned was about like
[26:56] context preservation. In our first version of query expert we just returned all top five context chunks back to the agent. But we found there was like greater hallucination rates and context window rot. So we got smarter here. So we always return the full content of the
[27:11] top two chunks based on that adjusted score. Then third place or lower items only return truncated or summarized context with explicit tools and direction for the agent to research and fetch that chunk. So we make it aware that there might be some more relevant
[27:28] uh information in the corpus, but you don't have to overwork your brain just yet. Then we put an explicit guardrail limit on what context is returned at all. So for any chunk that has an adjusted score that is 10% less than the highest score, we shield that entirely from the agent.
[27:44] This increases likelihood that only the most relevant context of service agent and reduces token cost in our evaluations. actually made it more reliable and more accurate.
[28:02] As much as we optimize uh the retrieval and augmentation steps, uh the biggest gain and hurdle is really to keep our context accurate and fresh. Uh it's really difficult to balance where where we want to be fully automated and where do we want humans to sanity check and contribute directly. To help solve for this, we invest in automations but route
[28:18] everything in GitHub as a backed source that provides both transparency and acts as a collaboration platform for both agents and humans. So this is all ran through data bricks pipelines that detect changes and updates context throughout but then posts that change back to GitHub as a pull request on
[28:35] merge gets updated back to the index. This helps us keep contexts up to date across the database. So for example, if a table becomes deprecated, we mark that table as inactive. Then remove all the queries referencing that table and then flag any knowledge docs that mention
[28:51] that table. That last method, user feedback. This is something that we're actually in beta now and we're really excited about. Um we're logging user steering with in in any agent harness where people are kind of steering query expert and say you use
[29:07] the wrong table or use the wrong column here. We're logging that and then feeding it back into our corpus. In practice, this is what it kind of looks like. We get we get about 15 to 20 PRs a week. Uh these are made by both bots and humans. And the really nice
[29:24] thing about going to like a file system with GitHub is that people were able to see and control the context within query expert. It really created more trust into our system and got more uh contributions throughout our enterprise.
[29:43] So query expert today. So we launched this, we originally architect this last spring and we launched it in July 2025. Um and we've seen our users grow. We now have about 3,500 people a month using query expert. Um part of the growth is a lot of people a lot of people at Block
[29:58] are becoming more AI native and AI first. But part of the growth is also putting the tool where users are. uh users are using it through their favorite agent harness. Uh ours is goose but it works in codeex, claude, you name it. Uh we also have a slackbot
[30:15] integration and users are building uh data dashboards using our block apps internal tool. So by bringing the capability to where users already are, we're seeing more people use it and more people getting value out of our data agent.
[30:31] All right. Next, I'm going to talk quickly about how data scientists uh can use Query Expert to do end-to-end analysis. So, not just answering ad hoc questions at this point, but doing a full-on deep dive. Um, so this slide shows the mental model for how I might work with this
[30:46] tool. Um, so one of the primary shifts in framing is that I want to give my agent as much context upfront about the whole analysis I'm trying to do rather than uh trickling in incremental ad hoc questions that don't share each other's context. Um, so initially I might
[31:02] provide my agent with a big brain dump to onboard it to what I'm trying to do, what questions I'm trying to answer, or what recommendations I'm trying to make. After the agent ingests that context, I'll do a broad round of data discovery with a sub agent using query expert. That process will save an artifact that can be passed through to any future
[31:17] sessions as needed and kept up to date or corrected. From there, we go into a planning phase where the agent will lay out a specific set of planned analysis topics and questions. Reviewing this analysis plan is a key high leverage moment for the human in the loop, confirming uh that
[31:33] reasonable data sources were discovered, the data actually exists to answer the questions in the plan, and the plan is aligned with the original intention of the analysis. After we're aligned on a plan, we'll fan out and do an analysis loop, which also leverages query expert, using sub aents again to do each individual analysis
[31:49] topic, saving artifacts that include SQL files about what was run and how, and then also saving individual markdown files for each topic that was analyzed by each sub agent. From there, we would fan back in to synthesize our results, potentially create visuals, readouts for different audiences as needed. And this
[32:05] review phase is again a key moment for the human in the loop. So, this isn't quite the do my entire job for me button yet, but it's safe to say that getting the vzero of your analysis has taken the turnaround time from days to hours using this tooling.
[32:23] Coming towards the end here, we want to acknowledge some limitations, challenges, and learnings and things we experience still day-to-day as we work on improving and iterating on this tool. First of all, to acknowledge query expert is not the only data agent at block for focused and repeatable workflows. There's absolutely use cases
[32:39] for more specific and bespoke agents that don't need to rediscover their data sources every time. Additionally, for business intelligence and reporting use cases, we have a dedicated governed metric store MCP that connects to our agents harnesses. We don't really want our CFO asking this tool how much revenue we had last week. Query expert
[32:56] is a complement and a building block to this other tooling, not some universal data solution. So, as with any agent, um context is how it feels smart. And we found that again keeping context up todate is the stickiest part of this problem. Um and this uh process we
[33:13] showed around autogenerated documentation and kind of scraping our query history is incredibly powerful and scalable but also can be fragile and the implementation and interpretation needs to all be done with care. um for contributing and reviewing to this documentation. Like Phil mentioned,
[33:29] we're still optimizing where exactly to put the human review in the loop and also learning that kind of different teams and different people would prefer to be in different parts of that process depending on how their own team operates across our very large diverse company. Next, supporting multiple personas is a
[33:44] challenge. I think we're doing it pretty well today. Um but a lot of this could be driven by people optimizing how they want to interact with this tool in their harness itself. you can give it instructions about uh how much you want to be reviewing, how much you want it to just run end to end. Um but needless to say, different personas, different types
[34:01] of people fulfilling different roles want to interact with this tool in different ways. Finally, for doing nuance analysis, like I showed on the previous slide, there still definitely is a need for a human in the loop. Um this is inherently a non-deterministic tool. I would say this almost leans into the non-determinism as
[34:17] a superpower. Um but still to produce really uh solid results in things that we trust it does require some amount of data literacy. Um inquiry expert should be thought of as kind of the the path towards implementing deterministic tooling andor human reviewed analysis more quickly. It's not again the full do
[34:34] my analysis for me button. Cool. All right. That's the end of our formal slides here. We're we're going to demo one or two quick things in our actual in goose here. Yeah. Let me see. Let me fire this up.
[34:51] All right, cool. Um, we'll go live. Yeah, let's try it. Prompt here.
[35:06] All right. So, we're here in Goose and just kind of like talk about Goose because this might be new. Um, this is our intern version of Goose. We have like our multi- aent workflow, a skill marketplace where I can run automations, um, schedule jobs. But here we have a project and here I can uh just tag query
[35:24] expert. I could dynamically call query expert, but I'll call it just for the sake of this demo. And I'm going to say like how many sellers edited 10 plus time cards in March 2025. Um, so you're going to see it spin here.
[35:40] Um, it's going to be looking across those databases, uh, come back and do that research for me and come back with a query, but also that deep uh, analysis information that we would expect this tool to have.
[36:05] So great. So this is the uh correct query. It's actually running the analysis for me. Great. It gives me a notes on logic of how it got there. And then this is where we talk about a data science co-worker. You know, want me to break it down further,
[36:21] you know, by audience or segment or cross reference against payroll merchants. How how did it know of all that stuff? you know that time cards was related to payroll or that uh this is most that what FNB means to square because that's like a very square lingo
[36:37] thing that means a food and beverage seller like a coffee shop or restaurant um it gets that all from our knowledge databases um so I can say like
[37:02] segmentations used recently. And so now it's going to like really find like, hey, what are like the most up-to-date segmentations that people have actually been caring about uh recently? Um, and it's going to come back with some ideas about what kind of segmentations to break down uh these
[37:17] numbers by. And I'll give you uh let me see I'll go through like some example tool calls. You know it's searching the knowledge and time card info. It's looking at the
[37:34] past queries kind of like we mentioned and see if it comes back with anything yet. So again is a data science coworker. It has all our segmentations baked into it.
[37:52] even like really deep dive stuff about people taking breaks or you know time cohorts or trends uh gross payment volume size um and audience and vertical I can say and then we talked about labeling that query um so this is a way
[38:09] that data scientists can just label the query query expert knows how to do that and what that means so I can say if it's something that I really like and I want it to be uploaded back to the uh corpus label that original query.
[38:25] Um so this will apply that query expert label that gets fed back into our knowledge corpus and then then that query and that metric gets can be updated uh for anybody to use and find.
[38:45] I think we're uh do you want just go back to one slide real quick? Um, yeah, I can kind of demo one nuance question just in the screenshot. Yeah,
[39:01] sorry. These are backups in case the live demo didn't didn't work too well. Um, but just for an example of a harder analysis question that it handles perfectly fine. Um, we could say here I ask it something about the square payroll products in a really specific slice being in Wyoming. How did it grow between 2021 and 2022? measure growth in the way that Square payroll measures
[39:18] growth. I don't want to give it anything specific there, pretending I kind of don't know anything about this topic. Um, and then also benchmark it against other states in the same geographical region. Again, I'm not trying to work too hard here. I don't want to give it a list of states. I wanted to just come up with all this on on the fly. Um, in this case, I also told it to save some
[39:34] artifacts, uh, being a markdown file and SQL file for what it did. So, here's just kind of what this comes back with after about one to two minutes. In this case, it takes a few more iterations. Um, but it tells me uh happily about how payroll grew in Wyoming in these two years. And then it'll also kind of flag
[39:52] specific things about the uh files it saved. It'll also flag that this is a really really really small client base and we shouldn't really draw any strong conclusions here, which is all fair. Um, but it's just a good demo of how kind of specific and bespoke you can get with this sort of tooling really quickly.
[40:07] Uh, that's all the time we had. Um, uh, thank you for coming to this talk and, uh, you know, we'll, we'll hang around a little bit afterwards if you have any questions for us. Thanks,

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.