Skip to main content

Curating a Genie Space

Summary

  • Six-step Genie curation: This video outlines a structured process to curate production-ready Genie spaces that deliver accurate, business-specific answers through data governance and configuration.
  • Metric standardization: Genie allows defining metrics one time using SQL expressions and certified metric views, ensuring all analysts use the same business logic and eliminate calculation debates.
  • Serverless compute: Genie spaces load with serverless warehouse compute, eliminating infrastructure setup complexity while maintaining governed data access through Unity Catalog for tables, metric views, and external federated data.

Curating a Genie Space

Watch: Curating a Genie Space
In this video, you will learn the six steps to curate a Genie space that delivers accurate answers tailored to your specific business needs. By following these steps, you can ensure your Genie space is ready for users and can be trusted.
🎯 What You'll Learn:
How to ensure you have the necessary data and that it is well-documented and governed by Unity Catalog
How to create and configure a Genie space
How to enhance Genie's understanding of your business language
How to define table relationships, metrics, and common business terms using SQL expressions, SQL queries, and SQL functions
How to monitor and evaluate Genie's accuracy by testing your space
How to certify your curated and tested Genie space
⏱️ CHAPTERS:
📚 Resources:
Learn more about curating a Genie space: https://docs.databricks.com/aws/en/genie/set-up

Chapters

FAQs

What data preparation is needed for a Genie space?

Data should be stored in your lakehouse or federated from external databases like Snowflake, all governed by Unity Catalog. Tables need descriptions, primary/foreign key relationships, and metadata so Genie has context for accurate answers.

How do metric views improve Genie accuracy?

Metric views standardize definitions once instead of having analysts calculate metrics differently. This creates centralized governance and ensures everyone gets consistent answers using the same certified business logic.

What is LakeFlow Connect's role in Genie setup?

LakeFlow Connect provides point-and-click ingestion from hundreds of sources without code. This prepares data quickly in your lakehouse for building and testing Genie spaces.

How are Genie spaces certified for production?

After teaching Genie your business language and defining relationships, test accuracy with benchmarks. Once validated, the space is certified and ready to share on serverless compute with no infrastructure management needed.

Full transcript

0:00 Introduction: Curating a Genie Space

Hi, I'm a senior technical marketing engineer here at Data Bricks. Do you want Genie to work for your specific business needs? Well, in this video, I'll walk you through six steps to curate a Genie space that delivers accurate answers tailored to your business needs. Step one, make sure you have access to the data needed to create your Genie space. The data can be stored in your lakehouse or accessed by federating into an external database.

0:20 Step 1: Ensure Data Access and Governance with Unity Catalog

Either way, the data and access controls are governed by Unity catalog. So whether your data lives in data bricks or in external systems like Snowflake, you can bring it all together. Make sure your data is well documented with table and column descriptions so that when you do create your space, Genie has context about your data as well. Additionally, make sure that your tables have defined primary key and foreign key relationships.
If you need to ingest data from an application or database, please check out LakeFlow Connect, which is point-and-click data ingestion from hundreds of sources with no custom code required. It's the fastest way to get your data ready for Genie. Step two, create your space. For the purpose of this demo, I will show you a space that has already been created and walk you through how it was created. This space was created on a single focus topic, which is workforce management.

1:18 Step 2: Create and Configure Your Genie Space

There are tables and metric views in here about employees, terminations, compensation, and even performance. Metric views are important because they offer the benefit of standardizing metric definitions one time. Instead of every analyst calculating employee raises differently, you define it once here. No more debates about which calculation is correct. This establishes trust and reliability through centralized governance and certification.
Everyone gets the same answer because everyone's using the same certified metric. Back in the space, give the space a detailed name that correlates to the Genie space topic and a description that will help future users of the space understand what the space is used for and how they can benefit from it. You can also add sample questions based on questions that you know future users are bound to ask and Genie automatically suggests some sample questions as well based on the data you provided.
Lastly, every Genie space should be loaded with compute from a serverless warehouse so you do not have to worry about infrastructure setup. Step three, teach Genie more about your business language. This is where accuracy really takes off. First, Genie automatically inherits information from Unity catalog metadata that you provided. Table descriptions, column descriptions, all of it. So, you're not starting from scratch.

2:44 Step 3: Teach Genie Your Business Language

You can edit a column description right here in the space. For example, I want Genie to know that when talking about departments, it is departments, employees, and independent contractors, so they are not left out of the analysis. This gives you flexibility to provide space-specific context. Please know that you can also directly edit column descriptions in Unity catalog for organizationwide definitions. Step four, define table relationships and metrics within your Genie space.
If your tables for and key relationships are not specified in Unity catalog, you can define join relationships in your Genie space. This is helpful for more complex join scenarios like self joins or if you don't have sufficient permission to modify the underlying tables. SQL expressions provide a structured guided way to teach Genie about common business terms such as KPIs, attributes, and conditions. Think of this as a quick way to define metrics in place like percentage of employees with arrays.

3:31 Step 4: Define Relationships and Metrics

You can also provide context such as synonyms and further instructions for Genie when it uses any defined SQL expressions. SQL queries provide Genie with examples on how to handle any intricate query patterns and multi-step logic. SQL functions registered and managed within Unity catalog can be used to help Genie address any questions that require custom aggregations or complex calculations. For example, flight risk rate, which is a specific function that calculates the number of employees leaving at any given time.
Step five, monitor and evaluate Genie by testing your space. Ask Genie questions that you know that other users will ask and make sure that it answers accurately. If it doesn't mark it for review so that you can go and investigate why Genie is getting it wrong and make the changes by adjusting text instructions, join statements, SQL expressions, SQL queries and or function instructions. You may also need to adjust column or table descriptions as well.

4:49 Step 5: Monitor and Evaluate Accuracy with Benchmarks

As you continue to test your Genie space, you'll find that some of Genie's responses are negatively rated. You can create a benchmark to analyze the question. What percentage of employees have left the sales department? Make sure you add the expected answer so Genie can validate its responses against known correct results. This becomes part of your testing framework. You can run these benchmarks regularly to ensure Genie's accuracy stays consistent as you add more data or make changes to the space.
If you get a bad assessment, make sure as the creator of the space to investigate why Genie is getting it wrong and make any changes such as adjusting text instructions, joint statements, SQL expressions, SQL queries, and or function instructions. You may also need to adjust column or table descriptions as well. Step six, share your Genie space. Now that your space is curated, tested, and ready for users, certify it to indicate it meets your organization's quality standards and is production ready.

6:17 Step 6: Certify and Share Your Production-Ready Genie Space

Then configure permissions in Unity catalog to control access levels such as who can ask questions, who can modify the space, and who can manage it. Copy the link and invite your team to start exploring. Follow these six steps and you'll have a genie space your users can trust. Now it's your turn. Go curate your own genie space and see what your team can discover on Data Bricks.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.