Sponsored by: Snowplow | The Missing Layer Between Your Lakehouse and Genie: Customer Context

Summary

Snowplow delivers a validated customer behavioral context layer directly into the lakehouse by capturing digital interactions across websites, apps, and server-side systems in governed event schemas stored in Databricks through Unity Catalog.
Pre-built semantic models and curated user profiles plug directly into Databricks Genie's knowledge store, enabling business users to self-serve behavioral insights without requiring custom engineering or data team involvement.
Every validated interaction generates richer behavioral signals that sharpen Genie answers and AI agent decisions, creating a self-reinforcing flywheel where the context layer continuously improves personalization and response accuracy over time.

Chapters

FAQs

What does Snowplow do and how does it integrate with Databricks?

Snowplow is a behavioral data collection platform that places trackers on websites, apps, and server-side systems to capture every digital interaction in validated event schemas. It ingests this data in near real time to Databricks, where it is governed through Unity Catalog and made available as a behavioral context layer for Genie and AI agents on the Databricks Data and AI platform.

Why is behavioral data important for AI context in a lakehouse?

Behavioral data captures what users actually do — which pages they visit, what they click, and how long they engage — revealing intent and preferences that structured transactional data alone cannot show. Without this context, AI agents produce generic answers that ignore the user-specific behavioral patterns needed for accurate, personalized responses.

How does Snowplow connect behavioral data to Databricks Genie?

Snowplow provides pre-built semantic models that plug directly into Genie's knowledge store, allowing business users to ask questions about customer behavior in plain English without needing data team support. The semantic models and reusable skills described in this video represent the highest-ROI context investment because they are built once and improve Genie answer quality continuously.

What is the AI feedback loop that Snowplow describes?

The feedback loop works by streaming validated behavioral data into Genie and AI agents, which then deliver more personalized and accurate experiences to users. Those improved interactions generate richer behavioral signals that further sharpen future agent decisions, creating a cycle where the context layer becomes more valuable the more it is used.

Full transcript

[00:06] All right, I think we're going to get started here. Hi everybody. Thanks for coming out. I know there's a lot of swag to be had, so I appreciate you taking 20 minutes out to come listen to me. So, my name is John Malloy. I am a senior technical account manager here
[00:18] with Snowplow. And if you don't know a lot about Snowplow, we are essentially a behavioral data collection company. So, what we can do is we can put trackers on pretty much any
[00:31] application that you have across your business stack. Those can be websites, those can be apps, those can be server-side things. And we can collect data for you guys in a really well-governed format and ingest it into our Snowplow pipeline, and we
[00:44] can get that data basically in real time to a number of downstream destinations. Databricks obviously being a super important one for us, but if you have other third-party destinations, other warehouses, we're pretty flexible on where that data wants to go.
[00:57] We saw during the keynote today that one of the big things that is coming now that AI is proliferating everywhere is the idea that AI is great, but it's only as good as the context that you give it. And that's
[01:10] something that we at Snowplow have really embraced. We have always believed that behavioral data that you can capture allows you to get an understanding about your users, who they are, and what they want. And
[01:23] now it is not only important to have analytical capabilities to understand those things, but you need to be able to feed those insights back into your AIs so that they are able to be more performative and give answers that are better than just vanilla out-of-the-box
[01:37] large language models. So, that's what I'm going to talk a little bit about today. I'm going to take you guys through a couple techniques that have worked really well for us. And hopefully that I can convince you that investing in your context layer is
[01:49] something that you should be thinking about, and that behavioral data can be a really important piece of that context layer that you're going to introduce. So, before we go to that, if I can get the clicker to go.
[02:03] Oh. If I can get the clicker to go back. Uh maybe I should try to tell you a little bit about who I am and why you should bother listening to me. So, I have spent 4 years as a technical account manager with Snowplow helping derive value out of the behavioral data
[02:16] that we are helping our customers capture. So, from that perspective alone, I think I have a pretty good insight into the types of challenges that organizations have when it comes to understanding their users and how the behavioral data we can help capture can
[02:29] help answer a lot of really tough questions about who your users are and what they really want. I also have another angle where I believe that I have some relevant experience in building contacts as far as AI is concerned in that for 4 years
[02:41] previously before AI really came out, I was working as a data scientist trying to build some machine learning type uh workflows in mostly the US healthcare space and this is back before AI came
[02:53] onto the scene and ML was really the hot topic that everybody was thinking about. And as I was doing that work, you know, looking back on it now, I realized that a lot of the work that I was doing at that point in time is work that
[03:06] made me highly replaceable with the introduction of Genie and other AI things because during that time I was not spending a lot of time or I wasn't spending as much time building machine
[03:18] learning flows as I was spending time reporting on what those machine learning flows were and trying to explain them to non-technical business users. I would go to my boss's boss, my grand boss. I would go to our board and
[03:31] try to explain, "Hey, I built this machine learning flow. Here's what it does and here's the impact that we had." Now, if I had had a tool like Genie back in the day, I wouldn't have had to spend so much time milling around in our
[03:43] warehouse writing SQL queries to answer basic questions about what the machine learning workflows I had previously built were getting us as far as an organization. The problem with trying to have tools
[03:55] like Genie answer complicated questions like those is that a lot of times the answers that you're going for are really specific and nuanced. Tools like Genie are awesome and they're super powerful and they give you the ability to answer
[04:07] questions that previously I had to be answering myself, taking me away from my machine learning, you know, responsibilities that I was supposed or would have rather have been working on. But today, if I just take Genie and I
[04:20] drop it into your warehouse and start to ask it questions, it's probably going to do a reasonable job asking reasonably generic questions, but the second I start getting into the nitty-gritty and trying to ask it some more complicated
[04:32] or specific questions without giving it the proper context about my organization, things that are important, and what certain elements of my data mean, the responses are going to just start start to suffer. You're going to start
[04:45] to see that the answers that your AI gives you back are generic, they're sometimes going to not going to be super consistent because AI is probabilistic. So, the more that you can introduce context and that you can give Genie
[04:59] something to bite its teeth into and the less it has to guess about what your data means, the better and more consistent responses you're going to get and you are going to free up people like me from having to answer these sort of
[05:12] boring grunt work style questions and let us do the, you know, fun machine learning or whatever jobs that you'd rather be doing. Taking the time to do this is going to free up a lot of that time and it's
[05:24] going to be super impactful for you. So, what is the goal here? What are we actually trying to do by introducing this context layer? We're trying to reduce bottlenecks. We're trying to get the proverbial mes
[05:37] from having to spend time analyzing questions that can be readily answered with AI if we put enough investment into our context and give the AI the tools it needs to answer these questions so that I can be freed up to go off and do other
[05:49] things. And then the other thing that we're trying to do is we're trying to take me as a person and take me out of the loop entirely, so that any individual analyst or any executive or anybody who comes
[06:01] across our data has the ability to interact with these agents and get the same results back. Uh so, by giving and all that is accomplished with context. Making sure that you are taking
[06:14] the data that you have, limiting the amount of guesswork and ambiguity that the AI has to deal with, and making sure that your foundation is robust enough so that these agents have what they need to do things.
[06:27] So, what are some of the things that if I'm thinking about creating a context layer and giving my data some information that it's going to need to avoid some of the pitfalls that AI likes to fall into,
[06:39] what are some of the things that I need to do to make sure that I'm getting the highest quality responses out of the questions I'm asking. So, I would say the number one thing that we see and that we at Snowplow have
[06:51] kind of leaned into over the years is make sure that your data is governed. Okay? Um agents want to make you happy and when you ask a question and they don't know what the answer to that question is or
[07:03] how to guess get the answer to that question, they're going to guess and they're going to guess to give you a response that they think is going to make you happy and maybe it will until you realize that what it said is made up. By making sure that your data is
[07:15] governed and everything is written in black and white for the agent to read, you're eliminating guesswork, you're eliminating ambiguity, and making sure that even though the agents are following probabilistic processes,
[07:29] they're going to follow the same flow every time because you've told them explicitly when you're answering this type of question, you want to be doing these type of things and you want to be using these type of data in this way. Um by that same kind of token,
[07:44] you want to make sure that the data that you're giving these agents is curated and relevant to the questions that they're trying to answer. Um, a lot of times when I'm working with big clients, their initial instinct when they're
[07:56] capturing behavioral data is to say, "Let's capture every piece of data we can possibly think of and shove it in a warehouse cuz the warehouse can handle it. Agents are good at parsing over big corpuses of data.
[08:08] Let's just grab it because I might need it later." I would argue that that is actually the wrong approach because you want to be approaching your data collection and your context creation with end use cases in mind. You
[08:20] want to have at least some sort of understanding about what type of questions you think are going to come down the pipe and you want to be creating a data ecosystem where
[08:32] the jumps that the AI is making as it's parsing through your corpus to try to figure out the answer to the question you're asking, those leaps are logical and make sense because the more you can allow it to do that,
[08:44] the more that you're going to get consistent answers. And so, you can see consistent is the last thing I kind of called out here. You really don't want agents answering the same questions in different ways, which they will do if you've given them too much ambiguity because again,
[08:57] they're going to be making that guesswork if they can't figure out the answers. So, give them the tools that they need and set them up for success. Don't just throw your agents at a whole bunch of unstructured data and say, "Hey, agent,
[09:11] you're good at reading lots of data. I have this question, go figure it out." Cuz it's going to figure something out, but is that thing going to be the correct answer? I don't know. And potentially, you might not know whether that's the correct answer until much further down the line
[09:23] when you've already made a bunch of decisions. And at that point, you know, you're trying to backtrack, you're trying to understand where things went went wonky. You know, don't let that happen. Try to figure this out ahead of time. It's not a sexy proposition to be
[09:36] spending time governing and curating the data that you're going to need ahead of time, but the results you're going to get in the long run are worth it. And we've kind of seen that. Um talking a little bit about how
[09:48] Snowplow specifically kind of forces you into doing that upfront governance and cure uh data curation. One of the big things the biggest point on this slide is the one all the way on
[10:01] my right, your left. Validated event schemas. So, when we're capturing Snowplow data, one of the things that we leaned into very early on in our company's history was making sure that when you go to capture data, you need to
[10:14] define what this data that you're capturing is. We really kind of through technology force you away from the let's just take all the data we could possibly think of and dump it in the warehouse. And we pushed you more toward a you're
[10:27] going to be capturing data with some sort of use case, some sort of questions in mind. Tell us what that is, encode that data into schemas, so that by the time the data reaches your warehouse and your analysts are looking at it, it's going
[10:39] to be well governed. That is a lot of upfront work. It's sometimes not the easiest thing in the world to get started, but once you're starting, you can hand that data off to humans or agents who have no idea what this data
[10:51] is or where it came from, and it's all going to make sense because you've done the legwork ahead of time to make sure that that those events are validated and following the schemas that you've defined. From there, it's pretty trivial to be building out other really valuable data
[11:05] assets because you know all your data's clean. You can build behavioral profiles, that's something that a lot of our customers care a lot about. Uh you can add knowledge store definitions really easily cuz all the data conforms to your schemas upfront. And then, you
[11:17] know, obviously you can throw this into Unity Catalog or any data catalog that you want and make that play harmoniously with the rest of your ecosystem. And what you're going to find is that Snowplow data tends to be a really nice,
[11:29] easy to work with layer of your data infrastructure because the schemas were validated ahead of time and everything you captured is coming through and landing in data bricks or wherever the way you expect. And then obviously once you have that,
[11:42] Genie or any other AI is happy to take pick it up and run with it. Genie and other AI plays really well with Snowplow data because it has a very consistent structure and you put in that legwork.
[11:55] So, talking a little bit about kind of some practical things that we at Snowplow have seen work really well. And keeping in mind that this is very much in the context of behavioral data because that's what Snowplow works to capture.
[12:07] And you know, these are just a couple of examples of things that, you know, like everybody else, we're experimenting with AI, trying to figure out what works and what doesn't. These are some things that have worked really well for us. Uh so, you know, building curated
[12:19] profiles, I already said uh as a thing that works really well, but just because AI is really good at taking large amounts of data and compressing it down, doesn't mean you want to skimp on the
[12:32] pre-work. If you take the time to figure out how you want to aggregate your, you know, event level data up to things that happen in a single session or the lifetime value of a user because
[12:44] every business has slightly different definitions of certain metrics. My lifetime business value might not be calculated exactly the same way as your lifetime business value, but an AI might not necessarily understand the nuances
[12:57] between our two definitions of those metrics. So, instead, don't make the AI guess how you want those metrics calculated. Build the pure curated profile and tell it, now you've got a row in your warehouse
[13:09] that represents an individual user and you have all of the aggregated metrics right there for the AI to consume. You can imagine a world that's really cool where AI can just go, pluck that user profile out of your warehouse, go back
[13:22] to that user who's actively interacting with your apps, and you can have it have a lot of context about that person without the person ever having to tell it anything. And that's a really good user experience. That's a really nice way to personalize an experience that
[13:36] you've got. Whereas, you know, if you're just saying, "Hey AI, go recommend some shoes to this person cuz they looked at a shoe." you know, it'll give them some shoes. It's happy to do that, but it might not be as good as you want.
[13:48] Um the pre-work, skills and metadata. I would say from Snowplow's internal use of AI, this is the number one thing that has gotten value for us. Uh we have a culture kind of at our company where everybody is encouraged to build skills,
[14:01] and if you're doing something more than once, encode it into a skill and share it so other people can use it. Uh that works just as well when you're thinking about agents like Genie as it is for, you know, Claude code, you know, any
[14:13] language model that you're using locally. Trying to codify and skillify all of your workflows so that they're reusable and that if an agent does this thing a particular way, you can go back and understand this is why it did it
[14:25] that way because I gave it instructions to do that. Uh those artifacts are only good, and you can put those up in repos, everybody can share and interact. That's how you get more of your company adopting AI and make AI feel more like a tractable
[14:38] thing. Uh our organization has everybody using AI because we've made it really accessible for people and helping people to understand that skills and profiles are the way to go like to
[14:52] make that happen has made our internal organization operate much more effectively, and you guys I think can do that for your um organization as well. Uh streaming data. This one's a little
[15:04] bit wonky and a little bit um on in the experimentation phase for us, but you know, I mentioned before that Snowplow enforces data governance up front, and so by the time the data, which is still real-time data, is
[15:16] streaming through the rest of our pipeline, it's already been curated, and we know it conforms to those schemas. We found that agents are actually very, very good at parsing that streaming data assuming that it's really well governed.
[15:28] So, even though in the profiles bit I kind of said, "Don't you know, feed it just a bunch of data and ask it questions about it." You can feed it, you know, well-governed streaming data where everything is well-defined and makes a lot of sense
[15:41] and ask it to just churn through tons and tons of events and have it spit out reasonably good results. Um we're doing a lot of experimentation with that now trying to understand how far we can push this, but our results have been more positive than we
[15:54] kind of initially thought. We kind of thought it would spit out a bunch of jumbled messes when we fed it just raw streaming data, but because our data quality at Snowflake was so good, it it's doing better than we expected.
[16:06] And then, this is one that I really like. Um one of the big problems with AI, as I mentioned, is that it's probabilistic and you can ask it the same question and get different results potentially. I like to remove in my personal
[16:18] workflows as much of that ambiguity as possible and that means anytime I have an AI do anything, so if I have it going through Databricks looking up stuff for me, I'm almost always asking that, "Hey, take whatever query you used to get this
[16:31] information and spit that out into a script for me." I like to understand what that agent did and if I have that as a reusable script, then I don't need to worry about the next AI coming along and doing something slightly differently. I can just run the
[16:44] script again. And I can ask an agent to run that script again for me, but in that way I don't have to worry about slightly different scripts over time or I don't have to worry about a co-worker, you know,
[16:57] operating in a slightly different methodology from me. Trying to figure out how to get away from just interfacing with a chatbot and pushing toward how do I actually get reusable assets out of my interaction with AI,
[17:11] um can really help you kind of push into a space that where you're a lot more comfortable with the results that it's giving you. Um So, you know, here's just a couple of examples uh for each of these categories, things that you can kind of
[17:25] mess around with if you want to, you know, it's basically what I just talked about. And then the last kind of point that I will make here is that we kind of envision a future where there is an AI feedback loop. And a lot of
[17:39] organizations are at least doing a few of these steps. So, you know, if you're a Snowflake customer, you are capturing event level data, that's what you're doing, and we're hoping that that data gets a lot of value for you. And we're right now we're kind of on
[17:51] step two on this diagram where we're trying to help feed that uh data into agents and create that rich uh context layer, which is something that I hope hopefully we've convinced you is something that's valuable and worth doing.
[18:04] The third layer, which not a lot of organizations are doing right now, but we're super excited about and we see a vision where people are doing that almost constantly is giving agents that are active on your
[18:16] apps access to the context layer that you are providing and helping them augment the user journey and the user experience in real time. So, that the next set of events that
[18:28] that user generates comes from a personalized experience. So, that can be scary to do because you're essentially putting an agent into production, giving it context that you've generated
[18:40] over a period of time, and then you don't necessarily know what the agent is going to do. You don't know how it's going to augment that user experience. If you depending on how free rein you make it, it might try to
[18:53] change the look and feel of the site, it try might try to serve content that, you know, you might not want it serving. But eventually, the guardrails around AI and what it's going to be able to do live for your users is going to get better
[19:05] and better, and it's going to need this personalized context for each of your users. Because once you it has access to that, it's going to make every interaction with your app feel like a personalized experience and
[19:18] it's going to just the the feel for all of your users is going to be noticeable and you're going to see this starts to accelerate and you're going to generate more data, which is going to feed back into even better
[19:31] personalized experiences. People are just going to come away happy. I we think that's a really cool thing. We're excited to help people build that. Um if anybody wants to talk to me about that, I'm super excited about trying to build that. So, feel free to come up and say hi.
[19:44] Um and that's pretty much all I have. So, hopefully this was helpful. If you guys have any questions, uh feel free to come chat with me, but hope you have a great rest of the conference. Appreciate it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.