From Space to Scale: What to Do (and Not Do) for Secure, Fast, Production AI/BI

Summary

This video presents a practitioner-tested playbook for deploying production Genie Spaces, recommending narrow scope with a maximum of five tables per space, strong data metadata alignment in Unity Catalog, and persona-specific instruction prompts to drive clarity and performance.
Genie Spaces can be integrated into multi-agent workflows using a supervisor agent that routes queries across multiple specialized spaces, enabling complex cross-domain analytics that a single space cannot handle reliably.
Production-grade operationalization requires pre-joined and trimmed tables, parameterized SQL as materialized views for common queries, audit-driven monitoring, and CI/CD promotion through GitHub repos to manage Genie assets at scale.

From Space to Scale: What to Do (and Not Do) for Secure, Fast, Production AI/BI

Learn the essential, Databricks AI FDE -tested playbook for deploying production Genie. Start by scoping topic-focused Spaces that leverage Unity Catalog and knowledge-store metadata along with parameterized example SQL rather than free-text queries. This approach drives both clarity and speed.
Before scaling out, establish pragmatic, business-centric benchmarks using trusted assets and value dictionaries to measure what truly matters. Integrate Genie into multi-agent workflows through the Conversation API, Teams, and the integration into Databricks’ Agent Framework. Operationalize at scale with pre-joined and trimmed tables, audit-driven monitoring, and a lightweight review cadence.
We'll share proven do's and don'ts, common pitfalls to avoid, and a reusable checklist to accelerate your time-to-value, all grounded in Unity Catalog metadata and real-world enterprise deployments.
Talk By: Robert Martin-Short, AI Engineer, Databricks ; Yinxi Zhang, Sr Staff Data Scientist, Databricks ;

Chapters

FAQs

What is the recommended scope for a Genie Space?

Best practice is to start small with a maximum of five tables per Genie Space, focused on a well-scoped business use case and tailored to a specific user persona. This narrow scope improves accuracy because the terminology and data context differ significantly between user types such as software developers and business analysts, and a broader scope makes it harder for Genie to interpret queries correctly.

How do you improve Genie Space performance through data quality?

The most impactful improvement comes from enriching Unity Catalog metadata — specifically table and column descriptions — so Genie understands what data to use and how to interpret it. Cleaner, better-described metadata consistently outperforms prompt engineering or model tuning as a performance lever, and improving metadata also benefits all other tools and users accessing those tables.

How can Genie Spaces be integrated into multi-agent architectures?

A multi-Genie supervisor agent can route user queries to the most appropriate specialized Genie Space, enabling cross-domain analytics at scale. This architecture allows individual Genie Spaces to remain focused and reliable while the supervisor handles orchestration and summarization of results from multiple spaces.

What CI/CD practices are recommended for Genie assets?

Genie assets should be managed and promoted through GitHub repositories to support version control and multi-environment deployment. Combined with audit-driven monitoring and a lightweight review cadence, this approach enables teams to operationalize Genie Spaces at scale — and the video also references community tooling for automated Genie space management.

Full transcript

[00:08] Hello everybody. Welcome to our session today from Genie space to scale. Could you give me a thumbs up if you can hear me, okay? Okay, cool. So, my name is Yingxi Jiang. I'm a senior staff AI engineer at the AIFD
[00:20] team. If you're curious about what AIFD team is, feel free to stop over at our center station in the booth and they'll tell you more about AIFD program. And presenting today with me is Robert. He
[00:33] will do his intros later and a lot of the content in the session is prepared by our colleague Andreas.
[00:45] All right. I know everybody get very excited today hearing about the keynotes. Databricks is basically a Genie platform now, but unfortunately the session today is not about the all the cool new Genie products which is announced today is from the smaller
[00:58] Genie unit, Genie space. We want to talk about practice when you are trying to build Genie space to get data insight for business users and how do we build multi-agent
[01:10] multi-Genie supervisor agent from a single space. Starting from the smallest modular unit, a single individual Genie space, when try when getting started, there are some
[01:23] best practice we recommend. The first thing is always like start small and build gradual have very very focused narrow scope business use case and make
[01:35] it a iterative process. Um to gain confidence and some trust when you just get started, the rule of thumb is to have max five tables per Genie space and you want it
[01:49] to be specialized to well-scoped business case and tailored it to a given persona. The reason for that is the language and terminology used by say software
[02:01] developer will be quite different from a product engineer or a business user. So, that will dictate how you design your instruction prompts. And other than
[02:13] that, the data are always the fundamental layer of your AI and data intelligence. Making sure you have alignment with the semantic layer, have very clear descriptions in your table
[02:26] metadata so that the genie know what to use and what to find can perform better. Um to improve performance throughout the iteration process, you will you will find out that the best way is actually
[02:39] to clean to create cleaner and better data metadata.
[02:51] All right. A lot of people ask, "When I create genie spaces, there are so many knobs that I can tune. Where do I start and what do I prioritize?" If we were to give you a curation priority list, this will be
[03:05] a good like guidance to follow. Again, the the most important and critical thing is the metadata itself, your table descriptions, your column descriptions. Those are the first line of source of
[03:19] truth that genie use to figure out which table to use and which column to use and how do I join the tables based on their definition of primary and foreign keys. After that, you can define richer
[03:34] knowledge and information in knowledge store that includes how does you match up to entities in the columns and how to do example joins, what are the
[03:47] sequel expressions and some of the synonyms in the glossary and terminologies. The next step after you set up the first two is trusted assets, which includes
[04:00] both example SQL queries, which will be used as reference and inspirations when Genie generates SQL queries and trusted and SQL functions, which can be used as a more deterministic tool to call by
[04:12] providing arguments to the connected SQL functions. Text instructions should be the last resort you use when tuning your Genie performance. The reason is being text
[04:25] instructions in your Genie space should be more general. It's not specific to any table or column. It's basically the guideline of how your Genie should behave and it also extends to every
[04:39] conversation turn when you interact with Genie so that it get occupied in each of each turn of your context window. The pro tip here is if you are asking
[04:53] about uh questions that's irrelevant to previous conversations, always better to start fresh so it your previous conversation doesn't occupy the context memory and Genie will be able to
[05:07] perform faster and better with the new context. Once you have spent enough time developing your individual Genie space,
[05:21] um make sure to check those product production readiness checkpoints before you ship it into production. The most critical and important thing, of course, is accuracy itself, making sure you have
[05:34] benchmark in the space and making sure you run the benchmark in the lower environment and and hit and it has hit certain accuracy threshold before you decided to deploy it. And in the
[05:48] benchmark and and evaluation process, if you figure out there are some common patterns that's dominating the user query. For example, like 80% of the user query turns out to use some joins of two
[06:02] tables. Maybe a better way to refine the model, the data model as well as to optimize the Genie performance is to create a materialized view of the joint table in the space. So, every time
[06:14] a similar question comes up, Genie directly goes to material material view rather than creating the join on the fly. So, hardening the data model will help as well. When you are rolling this
[06:26] out in production workspace for a larger scale users, uh rather than the the the smaller pilot users, think about how to plan for throughput. Um try to refresh the
[06:40] documentation on what is the latest latest the rate limit on Genie API to see if whether that fits your needs. Some tricks to um plan for throughput um is to
[06:52] use a agent to orchestrate Genie and use some retry exponential back-off uh options, and then you can also route non-query to uh different process, so it
[07:04] doesn't occupy the production workload of the Genie API. Uh if you know you're going to hit a large concurrency and a high scale of users, um you can talk to your account
[07:17] team to request an increase of throughput for the Genie spaces. Deploying a Genie space will be the same as you deploying any other software engineering apps, making sure you follow
[07:31] the code-first approach. Um use Genie API to export the asset in a Genie space as uh code as configuration files, and um
[07:43] dep- uh deployable assets and then trying to promote the assets through GitHub repository through code via CI/CD pipelines using Databricks steps.
[08:03] I know we have Genie 1 now, which will be able to automatically orchestrate multiple Genie spaces within the workspace, but sometimes users might want to have a finer finer or more granularity in their controls or might want to fine-tune
[08:17] their instructions based on specialized and a tailored use case. In those scenarios, customized the a customized supervisor agent is still helpful. You can create a supervisor agent through Agent Bricks. Um, you can
[08:31] also like do a DIY approach by orchestrating and authorizing your own agent. Um, the way to routing the way to optimize the routing from a supervisor agent to sub agents is
[08:43] by providing clear descriptions from the tools and if you started from Agent Brick besides Genie sub agent itself, you can also have seamless integration to other tools like SQL
[08:57] functions or your your additional MCP server that can help enrich the content of the overall use case. Um, when accessing enterprise
[09:10] sensitive data, it is recommended to use on user behalf authentication. So, when querying the supervisor agent, it pass through user's credential together with the row level permission control, it got
[09:24] rails the the query pattern making sure you don't leak sensitive information on to users. It will make it will read row role data based on what kind of
[09:38] permission you have per user. Deploying your supervisor agent as a Databricks app will also give you a seamless UI for end users to test it out. So, those are
[09:51] the recommended approaches if you're developing supervisor agent on Databricks.
[10:03] Supervisor agent is the same as like any other agent or your individual genie. You want to make sure you have the evaluation and monitoring and a feedback loop in the system to be able to monitor the performance and then build
[10:16] improvement to do iterations. Um the recommendation is to have traces implemented and build evaluation data sets that can that can be iterated and monitored. MLflow have all the
[10:29] capability of tracing, evaluation, and monitoring on Databricks platform. Um do the monitoring and then have a audit and a regression testing loop uh consistently and making sure to scale
[10:44] the operation model as a rinse and repeat pattern. Um we've been talking quite a lot about best practices and um it's very hard to remember everything. So, uh after this, Robert will show some
[10:58] handy tools which can automate this and where you just put it in the repository, don't need to uh remember everything in the development stage.
[11:14] Great. So, my name is Robert. I'm an AI uh engineer at Databricks and I'll be presenting these slides on behalf of my colleague Andreas. So, Andreas is the author of this uh best practices genie evaluation agent. And really the core concept here is Yishi just told you
[11:27] about how to build one genie space with best practices. But what happens when we already have many Genie Spaces and we want to verify that they are actually following best practices? Or maybe we have a Genie Space that's deployed in
[11:39] production and over time we want to confirm that it is actually still adhering to the best practices. That's what this evaluation tool will help us with.
[11:52] So as we know, best practices are easy to read but hard to keep. And when it comes to Genie Spaces, many of the issues here occur when the instructions that you provide to Genie Space are in conflict with the SQL
[12:06] queries that you provide it as as examples. And that's very difficult to spot. Also, Genie Spaces drift over time. You could have a perfectly functioning Genie Space at the beginning of your project, but then the underlying data tables
[12:18] might change and so a query that works well might end up not working so well and you'll need a subject matter expert actually querying with the right question in order to even spot this. And there's also a scale and ownership gap as well.
[12:31] supporting many Genie Spaces, um they might drift over time as I mentioned and the teams that are maintaining the tables underlying the Genie Spaces are different usually from the teams building the Spaces and they're different from the people who are actually querying it. And so we need to
[12:45] be able to quickly assess whether these Spaces are actually conforming to all the best practices that you just saw in the previous part of the presentation. So that's why we have this best practices evaluation agent. It's an open
[12:58] source project which I'll show you the QR code to download in the next slide. So what is this? Basically, it's a tool which assesses a Genie Space configuration and the configuration of the Unity Catalog tables that the Genie
[13:11] Space is working on. And that's important because the problems that typically happen are kind of at the interface between the instructions you give to the Genie Space and the tables in the catalog. So it will check the Databricks Databricks best practices, which is just
[13:24] in a markdown file in the GitHub repo, and there's also another file of your organization's best practices which you can edit. So, this reasoning model inside the agent framework will just assess the Genie space configuration
[13:36] based on the best practices document and the idea is it's going to produce a concise prioritized report with some to-do items which the team who's building this can then use to improve the Genie space.
[13:48] And if you're using this, I would recommend that you actually take a look at this report, implement the changes, and then run the evaluation again. Make sure the changes have actually been implemented correctly. And you could also imagine running this at scale on many Genie spaces and
[14:01] consolidating the reports using another summarization agent or something like that. So, what are these ideal best practices? They're basically all of the things that you should just talked about in the
[14:13] previous part of the presentation. They're here in this markdown file which is in the GitHub repo for this project and they're also available on the Databricks documentation online. So, what happens when the agent runs? It will just
[14:26] pull the latest version of this set of best practices so that it's not out of date and it will use that in its context when it's doing its evaluation. And this thing covers generic things like the Genie space design, modeling instructions, benchmarks, that type of
[14:38] thing. Now, when this tool was originally built, that was all of the best practices that we added. But over the course of field engineering deployments, we realized that customers have very
[14:50] specific policies and conventions that they want Genie spaces to be able to follow. So, for example, the definition of Q1 might be different between different organizations. How people want the Genie space to
[15:02] format things like percentages and currency and things like that. We need to be able to define that within the Genie space and we also can define that in the validation instructions. So, that's where all of this information goes
[15:15] inside the policy and conventions organization specific markdown file. Okay, so how do we use this? This is very simple to use. It's an open source uh project. You can scan this QR
[15:29] code and and get the repo. So, first thing you'll do is clone the repo to your uh to your Databricks Git folder in your workspace. You'll open up the uh notebook called run_validator. Uh you'll pick a space to analyze. Just
[15:41] provide the uh the ID. If there are benchmarks in the space, which we definitely recommend that you add, you can tell it whether or not it should be running the benchmarks. If so, it will run the benchmarks and use the results as part of its validation.
[15:54] You can choose the reasoning model from one of the many Databricks Foundation Model endpoints that are available to you. I think by default it's Claude Opus 4.8, but you can change it and see what results come out. And it will just produce this prioritized markdown
[16:06] report, which you can then share with your team. So, with that, let's go to a very brief demo. So, I'll show you how this works. So, let me go to this video.
[16:23] All right, so here we are inside the Databricks workspace. And we're going to run the the run the validator. You can see here we're choosing uh the different the space name, the space ID,
[16:35] and there's other things here. Just to make sure that we're setting up the configurations correctly. Under the hood, this is a LangGraph agent. So, you can see we're importing uh chat_databricks from uh from from
[16:48] LangChain there just to make sure the endpoint is working. And then we're running the validator using these configurations, and you can see that it runs. So, while it's running, let's take a look at what's actually going into it. Here's the evaluation reference guide
[17:00] that contains all of the Databricks supported best practices. Again, this is just available online. Here's your organizational convention best practices. Again, just a markdown file. You can leave that blank if you're just getting started, or you can fill it
[17:12] with as many best practices as you want. Next step is when it's doing this validation, it's going to run the Genie export API and grab the Genie space
[17:25] configuration as a JSON file. So, this is all of the information that the Genie space is using to generate its sequel and the validator also needs that. The other thing that it's going to do is grab a snapshot of the data tables that the Genie space is accessing. So, things
[17:38] like the schema, example values, so that it, as the validator, knows what the data is actually representing. And it's going to use both of those things. Now, let's go back to the validator to see it finishing up its running.
[17:52] So, it's done a bunch of things here. It's run the benchmarks. It's compared the configuration against several of these key points that it has inside its prompt. And now it's just basically going to produce this prioritized
[18:04] markdown report with this with with a set of instructions for your team to improve the Genie space. Let's take a look at what that report actually is. They always have this executive summary, which just explains the overall results.
[18:19] Then there's a set of critical issues, which you can step through and right try to resolve. And there's also a set of recommended improvements as well. So, you can imagine you can follow these and then you can run the validator again, see what markdown report comes out, and
[18:31] try to improve the situation like that.
[18:52] Okay. So, that's really the end. Please try it out on your spaces. As I said, it's open source. You can clone it. You can add your rules. Contribute to the repo if you want and let us know if you have any feedback. So, with that, thank you again for your attention.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.