Sponsored: Neo4j | Teaching Genie to See Connections Graph Intelligence for the Databricks Lakehouse

Summary

Neo4j extends Databricks Genie's analytical capability by enriching Delta Lake tables with graph-derived features—such as community detection scores and centrality metrics—that Genie can query as ordinary dimensions without changing the analyst workflow.
Flat SQL cannot identify multi-hop fraud patterns that only appear two or three relationship hops away from a suspicious transaction; graph algorithms run on Neo4j's Graph Data Science library surface these hidden rings and write computed results back to Delta Lake.
This video demonstrates the full pipeline for financial crime investigation on the Databricks Data and AI platform: from a raw transaction graph through community detection and centrality scoring to Genie producing relationship-aware fraud explanations.

Chapters

FAQs

Why can't flat SQL detect complex fraud patterns in financial data?

Multi-hop fraud patterns—such as money laundering rings that route funds through three or more intermediate accounts—are invisible to SQL JOIN operations because SQL is designed for direct table relationships, not traversals across chains of connected nodes. Graph algorithms can traverse these hops efficiently and identify clusters of suspicious activity that would never surface in a flat query result.

How does Neo4j enrich Delta Lake data for Genie to query?

Neo4j runs graph algorithms—including community detection and centrality scoring—on the transaction or account relationship graph, then writes the computed scores back to Delta Lake as additional columns. These graph-derived features appear as ordinary numeric dimensions that Genie on the Databricks Data and AI platform can reference in natural language queries.

What does the Neo4j and Databricks partnership include technically?

The partnership includes bidirectional data pipelines between Neo4j and Delta Lake, MCP server integration for AI agents, Agent Bricks connectivity, and a Virtual Graph capability that allows graph analysis to run directly on Delta Lake data with zero additional ETL—letting teams add graph intelligence to existing Databricks deployments.

What is the largest enterprise use case for graph analytics?

According to this video, the largest graph analytics use case is anti-money laundering and fraud detection, where mapping transaction networks and identifying suspicious account communities at multiple relationship hops provides detection capability impossible with relational data models alone. The same patterns also apply to pharmacy fraud and other industry-specific fraud scenarios.

Full transcript

[00:08] Welcome everyone. Thank you for coming out. Um it's it's great to see so many people here and I was wondering if how many people would actually show up at like the the morning after the concert. So it's thank you so much for coming out. Who here's worked with a graph
[00:21] database has a little bit of experience with graph databases? About half the audience. Great. Phenomenal. Just to give a little bit of background of what exactly is a graph database and what we're talking about today, a a graph database is a data model that I think of
[00:34] as a digital twin of the real world. And for example, you could do a data model of a map by mapping the cities and the roadways and how they interconnect and showing which cities are the most popular in the roadway by showing which
[00:47] what traffic flows the most through that those cities. Uh up here I have an example of an account. The account in the middle is showing how that account connects to other accounts and to other merchants. And so we're going to see how
[00:59] we can use this kind of graph model to do graph analysis. So the the graph model is works really well because it models how businesses work and allows you to do a data model
[01:12] that matches your business and how your business flows. Um, as an example, you can model like employees and their skills and their managers and how those interconnect. Uh you can map you can use suppliers and
[01:26] do supply chain visibility and see where part how parts are delivered in your supply chain and identify bottlenecks in your supply chain. Um transactions which we're going to be talking about today is one of our
[01:39] largest use cases and the biggest use case there is anti-moneyaundering and fraud.
[01:51] analysis uh pharmacy fraud. So these same patterns are applied across a wide variety of industries.
[02:03] So what we'll show today is teaching genie to see patterns in our data. And by teaching genie to see patterns or data what we can do is we can bring graph intelligence to the data bricks lakehouse. And so what we'll do is we'll enrich the data that's available to
[02:16] Genie and use leveraging that graph data. So to introduce myself, I'm Ryan Knight. I'm a partner architect for Neo forj focused on data bricks and I do two things. One is I build um the
[02:30] integrations between Neo Forj and data bricks and the second thing I do is I help customers in architecting their joint solutions between Neo Forj and data bricks.
[02:43] So data bricks and neo forj have an extensive partnership. We have uh ex a lot of different ways in which we interconnect and work together. The one thing we the the main connection we have is called our spark connector
[02:56] which runs in a traditional spark cluster and this creates a birectional data pipeline. So this allows you to extract your data out of the lakehouse and into the graph. It's like a
[03:08] materialized view of your data. Uh we have MCP servers for agent bricks uh and neo forj agent integration. We have agent memory for agent bricks. Uh so a
[03:20] lot of the use cases we're seeing is graph rag uh and graphenrich search for enterprise agent context. We also have unity catalog connector with the neo forj JDBC driver.
[03:33] And then we just announced a couple weeks ago virtual graph which allows you to do uh zero ETL graph analysis on your lakehouse. And how it works is you essentially create a map of your
[03:47] lakehouse data a schema map of a graph and then it takes that
[04:03] So the example I'm going to show today is how we can find fraud with graph and genie and how these two worlds can can be combined and work together.
[04:16] The example I'm going to be using is uh financial fraud detection. And as an example of how these the lakehouse works with the graph is you can imagine the lakehouse is your raw data source. All of your data flows into your lakehouse
[04:29] and your lakehouse will always contain the the raw data. Uh and you know lakehouses are really good for you know very for a lot of different use cases. On top of that what we want to do is a
[04:41] materialized view of a portion of our data. So we want to extract out the what is fits in the graph in a graph data model which in our example we're going to show accounts how those accounts
[04:53] relate to each other and then how those accounts relate to merchants. And the nice thing about this graph data model is from a graph you can easily surface fraud patterns such as two accounts are connected to the same address. You can
[05:06] model in your graph data. You could model like a social security as a node and so those nodes become a then you can show that same social security number maps to accounts. Uh to
[05:18] explain a little bit more about the graph what we have is a graph data model. It's called a labeled property graph. So each and the fundamental patterns in this are accounts are your
[05:30] nodes. So these are the entities or the nouns in your rel in your data model and those have labels which is like accounts or addresses merchants and the other
[05:42] data modeling tool we have is relationships. So the transfer 2 is a relationship between account A and account B. And because it's a labeled property graph, we put labels on the relationships. So we label all the
[05:54] entities and the relationships. And then it's called a property graph because we can attach properties to all the nodes and entities and all the relationships.
[06:12] So the motivating example I want to show today is how we can find financial crime using the a graph database with Genie. And the challenge with financial crime is it hides in the rows. So we can't
[06:26] just take a normal SQL queries and say all right give me the highest volume of transactions which accounts uh you know are transferring the most money or which accounts are you know sending the you know work with the merch these merchants
[06:39] the most commonly because those could be valid use cases. Those could just be someone that's moving a lot of money. They could be popular merchants and so rows in isolation and transactions in isolation will often look very clean.
[06:54] The challenge is that the fraud is hidden in the patterns across all these accounts. And so instead of looking at individual transactions and how these transactions like sum or averages of these transactions, we want to look at the
[07:06] patterns of these accounts and how they transact. So the role level aggregation cannot find those patterns. Uh and what we'll see is if you had just
[07:18] a traditional relational data model, the questions I can ask Genie are very limited. I can say which merchants are most commonly transacted with by the top 10% of accounts. So I don't have a fraud signal I can talk to Genie about and
[07:32] Genie will happily answer what is correct. All right well these look like the you know the top 10% of accounts by total dollar amount but this is not fraud. This is just normal transaction volume and so I don't have a clear signal of what is fraud. What we want to
[07:46] do is we want to be able to say, "All right, Genie, which merchants are mostly commonly transacted with by accounts that are in ring
[07:58] signal. And then what it can do is it can surface what are actually fraudulent accounts, potentially fraudulent accounts. And so it's able to get teach Genie new information.
[08:12] >> So what I can do is I can do better data for better Genie answers. So the core value of this is graph analysis with graph patterns. So graph only needs to describe a pattern. So SQL
[08:25] you start with a row or ids and I need an entry point into that and so I you know so I say you know summarize these rows give me these averages. I can't describe a general pattern and say you
[08:37] know from this lakehouse give me this pattern of data. And so traditional SQL is you know operates at the row level. What we want to do is operate on the network as a whole. And we want to look at how those accounts and those
[08:50] merchants are interacting and surface fraud. And so one example is centrality. How central is an account to the flow of a money. So you might see, you know, you have, you know, a network of 20 accounts
[09:03] and you see that they're all transacting through the same single account and that account always takes that money and transfers it to a single merchant. So now I'm seeing a pattern of okay, everyone's using this single node as kind of a central hub for
[09:16] the moneyaundering. I can see community. So which accounts cluster tightly together. So community detection is interesting because it's not just like which accounts are transacting with which merchants the most often. It's all
[09:29] right these of these patterns of you know like four or five accounts. They all transfer across each other and they all have the same patterns of transfers between each other. And so we're surfacing patterns of those transactions instead of actual rows.
[09:44] So really if you think about a fraud ring what a fraud ring is is it's really a subgraph. And so if we to think of our overall accounts and merchants, what we want to do is we want to find the subgraph inside of our overall graph
[09:57] that shows what is a fraud ring and label the data with that fraud ring. So finding the ring means finding the the cluster accounts moving money densely amongst themselves and what you know patterns within our network that look
[10:09] fraudulent. And this is our what I'd call like structural signals of fraud and these indicate the ring light behavior. So no so no road level aggregation can produce a network property. I can't see these patterns and
[10:21] label potential fraud rings with a SQL query. And so the way we can do this is we use an enrichment pipeline. And so this is how we enrich the the lakehouse. So
[10:35] there's really kind of four broad steps to convert a network of account relationships into plain columns so that Gina can then query like any other dimension. So first we're going to load the silver tables into Neo forj as a property graph. So your raw data flows
[10:49] into the lakehouse into a bronze table. From that bronze table it's often cleansed into a silver table and cleaned up in that silver table. And then from there uh we've loaded our property
[11:02] graph. Then we run going to run GDS. This is graph data science. These are our graph algorithms. And we're going to run and process the graph algorithms surface the fraud. And now our graph has been enriched with these fraud signals. I can then take this data and I can pull
[11:16] this data back down into a gold table and do a golden enriched table that now has the additional graph analysis. And I've now have a much richer data set that I can then use to query Genie.
[11:35] So graph columns really change what Genie can finds. So when I change the columns, I'm changing what Genie finds. And I'm I'm providing a much richer data set for Genie to act on. So the graph results enrich the gold tables with things like I can say like risk scores,
[11:48] community ids, similarity scores, how similar are accounts, uh fraud risks here. I could rank all my accounts using we have a a graph algorithm called page ranking. And Genie treats these like any
[12:00] other dimensions and it sees it like a dimension in our our lakehouse data. And so then I can do things like group by fraud risk tier where it is a r uh ring candidate and that's going to surface which of these ring candidates are the
[12:14] highest for fraud risk. And so this opens a whole new category of questions I can now ask Jeie about and talk to Jeie about how are how you know like candidate population sizing
[12:26] regional review workload. So you know which merchants transact with which you know fraudulent accounts the most. All right. And I'm going to show you a demo of this.
[12:38] So going back to our example, uh you see on the left hand side here, accounts are the the pink nodes, merchants are the yellow nodes. So this is showing the right this Rodrigo and
[12:52] how he's connected to these different merchants. We see in this here the relationships that you know of similar to who like here we see who Rodrigo's transacted with and who he's transferred to. And what I want to show is so what
[13:07] it's happened here is you know I have 32,000 notes I have 32,000 you know accounts and merchants and I want to show how we can take the this data and run graph algorithms on it and then load that back down into the lakehouse.
[13:21] So to do that, so this is really small. Um, what we do is we do a a projection. So
[13:33] what we're going to do is we're going to project a subset of the graph into
[13:46] accounts and make the orientation unidirectional. What's happened when I first loaded the graph is those accounts would be like account A transfers to account B. So I have a onedirectional relationship in my graph. I want to take
[13:58] that single direction unidirectional and make it birectional for analysis. So even if graph A or sorry account A talk to account B. I want to be able to to traverse the graph backwards and going
[14:10] from graph B back into sorry node A back into node B. Another sorry I'm saying this backwards node B back into node A. So I want to be able to traverse the graph in both directions
[14:27] and I should yeah I should have done that. So this is an example of like a page rank algorithm and this is how simple it is to to run these graph algorithms is I'm silly saying graph data science run a B page
[14:40] rank on that projection. I did a projection called account transfers. I want to iterate 20 times and write back out a something called a risk score. So a page rank is going to rank the nodes
[14:52] for potential risk based on uh those patterns of how they interact with other nodes.
[15:05] Uh the other example is what we call luane community detection. So again on that same projection I'm going to say run a Louvane community detection and cluster my nodes together by community ID. So let me go ahead and run. Let's
[15:20] see here. Run that. Of course it's not going to work.
[15:33] All right. So this is running the the graph community detection and it's it's basically going to cluster my network by these community ids. Of course it's it won't work today but anyways um and then what I can do is now that
[15:47] I've surfaced it I can run queries against it and say where this you know which of these accounts are grouped by the community and then do the community counts things like that and I can run queries against those.
[16:00] So this is as simple as it is is now that I've done this. Let me go back to this. Uh right. So I've enriched what I've done is I've written back out the property community ID on accounts. And
[16:12] so now my accounts have a new property called community ID. What I want to do is take that community ID and pull that back down to the lakehouse so Genie can see it.
[16:26] And so I have a a whole GitHub repository that I will share with you at the end and I'll put up on the screen. What it is is a series of notebooks that walks through setting up the connection to Neo forj. Setting up the initial properties. Uh then I'm run an initial
[16:41] like just analysis of Genie on basic silver questions. Uh, and then I ingest that data into Neo forj, run the enrichment, and then I pull those gold tables back down. And
[16:53] pulling the gold tables back down is doing things like just reading from Spark those graph features. And oh, let me move this out of the way. Right. And so now that I've enriched it,
[17:06] I've I I can read these graph features back down. And you see in the table like I have account ID, risk score. So now I have this data available and I've surfaced this data to inside of Spark. I take it from Spark right down into the
[17:19] lakehouse. now that I've done this, I can show you. So this is, let me go here. So this is
[17:33] Genie before I've let me scroll up so you can see. Yeah, let me
[17:45] So this is genie before right. So I've said I've created a genie on the date of transactions accounts. This is genie against the silver tables. And the best I can say are there accounts acting as hubs of potential fraudulent money movements.
[17:57] And if we look at this you know it's come up as some potential what it says is potential fraudulent. And then but it even says like well I I can't say exactly but I can just show you the the ones with like the highest
[18:09] balance of account transfers. What I can do after is I do that I I create a new genie space and what you can see is I've added the gold layer in there. So, I've added the gold accounts,
[18:22] the gold fraud rings, and now that now that Genie has additional signals, I can say questions like, are there accounts acting as hubs of potential fraudulent money movement? And it says, yes, there's these accounts are acting as
[18:34] hubs and it's able to draw a nice, you know, diagram top hubs by network centrality. So, it's it's surfacing these graph data science algorithms I ran.
[18:46] So um just to finish off this is just one pattern of how NeoRaj can supercharge your uh data bricks ecosystem in your your enterprise AI. We have a lot of other
[18:59] patterns. We have an entire graph uh uh entire graph intelligence platform. We enhance rag with what we call graph rag. So it's graphenrich search. Uh we do
[19:11] semantic mapping. uh agentic memory there is a lot of overlap after the keynotes this week there is some over so love to talk to you guys about that um context graphs multihop reasoning so
[19:25] love to talk to you guys more about all these patterns uh I'll be at the booth we have a booth over at 600 over in the 600 area please come by connect with us ask us questions I'm happy to talk about all these patterns and thank you so much
[19:39] everyone for coming

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.