Skip to main content

Sponsored by: Promethium | Delivering Self-Service Data for AI Scale on Databricks

Summary

  • Instant data products: Promethium generates contextual Data Answers enabling immediate, iterative data exploration at AI scale on Databricks.
  • Genie integration: Promethium's agent capability extends Genie with context from other enterprise data and applications to ensure accuracy and relevance.
  • Multi-source queries: The solution works across Databricks and other enterprise data platforms enabling agile, multi-shot questions and answers.

Sponsored by: Promethium | Delivering Self-Service Data for AI Scale on Databricks

Watch: Sponsored by: Promethium | Delivering Self-Service Data for AI Scale on Databricks
Talk by Prat Moghe, CEO, Promethium.ai (Sponsor Speaker)
AI initiatives often stall when data teams can’t keep up with business demand for ad hoc, self-service data. Whether it’s AI agents, BI tools, or business users—everyone needs data immediately, but the pipeline-centric modern data stack is not built for this scale of agility. Promethium enables the data teams to generate instant, contextual data products called Data Answers based on rapid, exploratory questions from the business. Data Answers empower data teams for AI-scale collaboration with the business. We will demo Promethium’s new agent capability to build data answers on Databricks for self-service data. The Promethium agent leverages and extends Genie with context from other enterprise data and applications to ensure accuracy and relevance.
Intelligent analytics for everyone: https://www.databricks.com/product/business-intelligence
BI meets AI: https://www.databricks.com/resources/ebook/business-intelligence-meets-ai
Explore the only unified platform for agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What makes Data Answers different from traditional data products?

Traditional data pipelines move data from source to destination in a single batch. Data Answers enable instant, iterative exploration that's multi-shot (allowing refinement), sharable and works on all enterprise data. This new experience is built for AI and business agility.

What problem does Promethium solve?

AI initiatives often stall when data teams can't keep up with business demand for ad hoc, self-service data. Promethium enables data teams to generate instant Data Answers based on rapid exploratory questions, empowering data teams for AI-scale collaboration with the business.

How can I give business users self-service access to data without relying on analysts for every question?

Giving business users self-service data access without overwhelming your analytics team requires building an intelligent data layer that understands business context and can automatically translate natural language questions into optimized queries. Promethium addresses this by layering on top of Databricks to provide semantic understanding of business data, enabling business users to ask questions in natural language while agents automatically handle query generation, optimization, and access control. This approach scales business analytics teams' impact by orders of magnitude because each analyst can now serve many more business users.

What are the best BI tools for self-service analytics?

Promethium combined with Databricks provides one of the most comprehensive platforms for self-service analytics because it brings together semantic understanding of business data, natural language query capabilities, and enterprise governance through Unity Catalog. While traditional BI tools like Tableau and Power BI offer visualization capabilities, Promethium's intelligent data layer and agent capabilities enable business users to get answers without predefined dashboards. The platform's strength lies in combining accessibility for business users with the governance and scale that enterprise data teams require.

self-service data access with automated approvals

Self-service data access with automated approvals removes the approval bottleneck that often limits analytics agility while maintaining security and compliance requirements. Promethium's agent-based approach can automatically assess data access requests against policies, approving routine requests instantly while escalating sensitive access to humans for review. This allows business teams to move at startup speed while maintaining the governance controls large enterprises require.

Full transcript

Hi everyone. Can you guys hear me? Awesome. Um, so I know we are between this and the cocktail hour or dinner. U nice nice to see you all. Uh, I'm the CEO of Prometheium. Um we're going to talk about um how to deliver self-service data um particularly in the in the world of AI and AI scale on data bricks uh and other enterprise data. Uh next slide. So again I'm also joined on the stage uh by who's our VP of engineering.
Um so again I'll I'll start you know it's very clear to everybody right now um with AI the business wants access to data fast so we see the need for not just the classic pipelines and data engineering we're seeing the need for more immediate iterative um data uh access to all your data uh for self-service so and when you think about what that requires um we are seeing that you know the current modern data stack which was built for pipelines um which was built on this idea for batch um which is singleshot so you start at a source and then you get at a destination uh and it's always on pre-selected data that doesn't work so you need essentially a new experience for AI one that is instant so when you want it you get it uh no you don't need lots of collaboration between humans.
Uh pipelines don't don't cut it as an architect. So you need a new experience which is more questions and answers. So it could be an agent asking a question and you get an answer or a human asking a question, you get an answer or an application or an API. Um it's got to be multi-shot. Sometimes the answer is not right and you want to iterate. Um it's got to be sharable. So whatever you produce as a response needs to be shared.
And then finally, it's got to work on all enterprise data. So if you have data sitting in data bricks with unity catalog, that's great. But you may also have an example where you have other data platforms. And so what you want is to be able to run those questions on all your data platforms including data bricks. So that's that's the new experience you want. And that requires a new architecture. Uh what this is showing you is that with Prometheium you can you can essentially u uh deploy that new architecture and it's an overlay on your existing enterprise.
It doesn't change anything. So data bricks can be on the left there where you have data sources with other enterprise data and on the right side you typically will have your consumption tools. So PowerBI, Tableau, Click you know whatever the BI tools or applications that you use. And then you may have a pipeline based ETL running today between all those data sources uh to be able to create dashboards and new data products.
So Prometheium comes and sits on top of this existing infrastructure. It doesn't change anything. It just gives you a new way for all the new add questions coming from the business. So with Prometheium your data teams it could be a data engineer it could be a data analyst or an analytical engineer you essentially use Prometheium get the questions from the business and when you get a question into Prometheium Prometheium will immediately query all your distributed data.
So it could be data in data bricks. It could be data in other data platforms or applications like Salesforce and it will on the fly build a data answer and that data answer then can be consumed into your existing tools. So you're not changing essentially the experience. Okay. So you're meeting the business people where they are. So how does this thing work? Uh next slide. So the data answer that Prometheium produces essentially is on the fly dynamically generated.
It's got all the context you need. So it has SQL code lineage on how this answer came from you know data bricks or other platforms metadata. You can chart it. Um and then the nice thing is it's an open architecture. So these data answers can be saved in platforms like data bricks. You know you can materialize them. You can save them as views or tables. Um, you can also feed them to other agents through MCP. Uh, Prometheium is an MCP server.
So, it's a tool. So, if you have any application that requires access to data, you can essentially make a call to Prometheium to MCP. We can also trigger new pipelines. So, if you said this data answer is good, I validated a hypothesis. I want to run it 20 times. U, you can actually save this as a DBT model and trigger a new pipeline as well. And then finally, you can publish it into all your existing uh BI tools. Uh so you're not changing anything.
So the key point here is these data answers are instant iterative. Um and they are essentially reusable because they're versioned. Um and so the next time when somebody asks a question, if you feel like this data answer is going to answer that question, then no query is done on data bricks or your back end. Basically the same data answer can be used. So it the agent has memory basically. Uh next slide. Um so this is just showing you how is this happening.
If you look inside Prometheium, there's basically three layers. At the bottommost layer uh is an instant data fabric which uses an open-source engine to basically go do federated querying across all your data sources and data platforms. So that could include data fab you know it could include data bricks it could include other data platforms on prem in the cloud uh applications like Salesforce um and then on top we have this 360 degree context engine this is the one that brings context in either it's technical metadata like in unity catalog can bring business context maybe from your BI tools or semantic layers or other cataloges um and it essentially brings that business and technical context text in one place and it builds that on the fly and then on top there is an agent called a data answer agent and this is the agent where your BI tools can point to it or agents can point it or or your data analysts can sit there and say here's a new question uh I want an answer to that and that's the agent that you use um and so also you have the option of a marketplace here the data answers can be published to a marketplace where again if you guys are Have you heard of data mesh concept where b different business domains have their idea of the things that they need?
That data mesh is available to you through Prometheium. So you don't have to build another layer. U so it gives you a marketplace where different groups can basically collaborate and say here are my questions and those can essentially be instantiated through the data marketplace and you can collaborate through this marketplace. Next slide. Um, so the key point of Prometheium is we're going to show you a demo, but but the key point is your small data teams can essentially very quickly be able to answer add-on questions coming from the business.
Um, it's a 10x faster capability for those add-on questions and it makes your teams more productive. And then obviously the more questions you answer and you share it with the business, this is a closed loop. The business can endorse those questions and then that will activate the metadata. So this is a smart system and essentially the more questions you answer the more insights you can uncover which leads to um you know business value basically.
Next slide. Um again this concept has been it's a new concept. It's been proven with large customers data teams uh like builders national grid quest diagnostics vital forms. We are also a cool vendor with Gartner uh where this new category of using a data fabric for self-service data is something that has a lot of momentum in the market. So happy to give you more information if you want to double click. So we're going to turn over um and then again we have many use cases where in large enterprises if you're in one of these enterprises which has data bricks but you also have other data platforms like snowflake or like Salesforce SQL server and many other applications you essentially can use Prometheium to generate data answers and then you can save those data answers in data bricks if you're a essentially that's your main platform and then you can consume them in a variety of these tools.
So we're just showing you examples of tools people are using whether it's PowerBI um whether it's Tableau or Hotspot. Does that make sense? So I'm going we're going to flip to a a quick demo so that you can see this a in action. So I'm going to turn it over to talk about what he's going to show. So can you guys hear me? So uh before I show the demo, here's what I'm going to you're going to see. So there's a concept of like a Prometheium agent.
So, Prometheium agent is a new UI on top of the data fabric and the context engine that we have. This is where you can ask questions, business questions, you can ask questions about discovery. You can ask questions about if you want to crawl any new data sources or anything like that. And that would use Prometheium's MCP servers or a set of servers so that you can get access to that information really quickly. The other thing I'm going to show you is not just using the Prometheium UI, but you can have your own agent. uh all of the all of you must be experimenting with various agentic systems.
So you can use that to uh talk to the Prometheian MC and CD servers. Data bricks already has this concept of the multi- aent system where genie agent can be one of the components. So if you are interested in using the genie agent to generate the SQL but what if you wanted additional context to be pulled from your external catalogs if you had external business cataloges like Alian or Calibbra or uh even semantic rules that are just tribal knowledge with in people's heads how do you give that information to the agent so that agent will so that the genie agent can actually pro produce more trustworthy answers so that's what you could do with with the bring your own agent and as I said you can also use a database genie agent to combine that.
So let me show you a quick demo of that. So this is the screen for our new agent we call it monitor. Um but you can ask questions about okay this is in this case we have some uh information about um retail sales. So that's the data that we have ingested. That's the context we have. And let's say if someone wants to ask a question about okay give me all the sales revenue uh from Georgia from say 2023. So you can ask question in natural language.
What this will do is go find all your technical metadata, go find all your context engine that you have and see if someone has already answered this question. So let's say in this case the data product uh may have already been created. um in which case you can choose from the list but sometimes what you're the question you are asking is completely brand new in which case you want to actually generate a new new question al together so in this case I'll go and click create data answer again in this what the agent is going to do is using your context engine is going to construct a SQL that will answer the question that you ask and it's not just the SQL as talked about you have access to the results the result set you have access to the SQL And more importantly there's explanation like because this is what is gives you the confidence in terms of how we arrived at the answer.
It's not enough to know the SQL is there but you want to know how we chose the tables why certain joints were picked why certain categories were used uh to answer this question. So that's why explanation is also very important. We also generate a quick lineage so that you know how the data is flowing from various systems. Uh so this is an example of how you can actually use the Prometheium UI to generate answers. Uh as I mentioned you can always create your own uh create your own agent and integrate Prometheium's agent with it.
So here um in this this example I have um we have information for a fictitious insurance company and I'm going to ask a question about who are the most profitable customers for this insurance company. And as you can imagine um every business has their own rules. Every insurance company has their own rules. So what's going to happen is the definition of profitable could be very different based on what your business rules are.
And sometimes these rules are defined in your catalog in your semantic layer and as I said sometimes just the users or the data product owners know what the rules are. So let's let's ask this question. So say top 20 most profitable This is going to invol this is going to invoke all the MCP tools that we talked about. So in this case it's going to find it's going to find all the relevant tables. It's going to find the business context.
As I said business context could come could come from many sources. It could also find any previously endorsed queries because that's again a powerful signal. previously someone had asked similar questions and if you have already answered with a specific SQL there's a good chance that it's a good fit. So we actually use that as a signal to improve the results and as you can see it actually tells you why it came up with this answer.
I talked about what is the most profitable u customer for this uh tenant. The definition is if the monthly financial stables has the profitable value as one more than or the loss ratio is more than 1. 0. That's the that's the definition that we have for this customer. And you can find that business context right here. And I'm just showing you some of the internal details on how you can debug these rules. And just and just like that just by asking a simple uh question you can get SQL the data product which can go across various uh data sources.
So this doesn't have to all the data doesn't have to be all in data bricks it could be in Salesforce your applications and still uh the agent will be able to get get to all to all the system. Another example that I talked about is genie agent. So if you add this agent as one of the react agents or one of the tools along with Genie then we can actually provide Genie with the contextual information so that you can still use Genie to produce the SQL but use the context that Prometheium has because the data is actually outside of data bricks.
Um also quickly want to show the various data sources that we support. These are all the data sources. In addition to these data sources, we also support many metadata sources. So you can actually grab this information from any of your systems and then enrich enrich the fabric which actually improves the quality of the results that you get when you ask these questions. This is the this is the instant data fabric layer on which if you remember there was a context engine and then the agent.
So what dures was showing you is that instant data fabric is the one that hydrates and builds that catalog and then the agent uses that catalog to basically answer those questions and if you are like administrative users where you need to see all the data and everything that's all always available uh to the so this is building the catalog actively on the fly and then basically it's smart enough to allow you to basically bring context from other sources into this catalog so it can take get leverage Unity for example, but if you have other cataloges in there, it can also bring information from those cataloges or semantic models.
Okay, with that we is that good? Yep, that's good. If you have any questions, we'll be happy to answer your questions. So, that was the end of the demo, but we are around. If you have guys have any questions, happy to answer them. Um, thanks again for your time. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.