Sponsored by: Ziggiz | The Security Lakehouse You've Already Decided to Build

Summary

Ziggiz is a security data lakehouse accelerator built natively on Databricks that solves the three hardest production challenges: high-volume security data collection, automated Delta Lake schema management as upstream sources change, and delivery of a detection-ready semantic model without months of data engineering.
Founding engineer Ben Newton, who previously built HSBC's cyber data platform on Databricks, demonstrates how Ziggiz handles upstream schema changes automatically and delivers a normalized semantic layer enabling complex detection at scale without manual schema wrangling.
This video targets security and data teams that have already committed to the lakehouse path, showing what a production-ready Databricks security data platform looks like—including multi-department data access and a fast path from onboarding to detection-capable production.

Chapters

FAQs

What problem does Ziggiz solve for security teams building a data lakehouse?

Ziggiz addresses the three hardest challenges in production security data lakehouse implementations: collecting security data at scale (terabytes per day from hundreds of sources), managing the Delta Lake layer without manual schema wrangling as upstream sources change, and delivering a detection-ready semantic model that works out of the box without a lengthy data engineering project.

Why is cybersecurity data particularly difficult to work with?

Security data is the digital exhaust of an enterprise estate—not clean reference data but raw, high-volume output from firewalls, proxies, endpoints, and hundreds of other sources. Sources change their schemas without notice, data is frequently missing or malformed, and the volume can reach terabytes per day at global bank scale.

What is Ziggiz's approach to handling upstream schema changes?

Ziggiz handles schema changes from upstream data sources automatically, eliminating the manual effort of updating pipelines and schemas each time a security vendor changes their data output format. This is built into the Ziggiz architecture as a native capability, reducing the operational burden on data engineering teams managing security data pipelines.

Who is the target audience for this video and what does the demo show?

This video targets security teams and data engineers who have already decided to build on the lakehouse path and want to see what production-ready looks like on Databricks. The demo covers the data ingestion workflow, automated schema diversity handling across multiple security sources, and the semantic layer that enables complex detection capabilities and multi-department access to security data.

Full transcript

[00:08] Good afternoon, ladies and gentlemen. Thank you very much for your time and attendance here. Yes, yes, Cody, get your headphones on. Are we sitting comfortably? Then we shall begin.
[00:21] Thank you everyone for coming. The fact that you've turned up here on the last day of the conference when you could be lying in bed with a hangover proves that either you are incredibly dedicated to improving your cybersecurity estate
[00:33] or you had nothing else fun to do. My name is Benjamin Newton. I am a founding engineer at Zigers and we solve a very simple problem. The problem of building a cybersecurity
[00:46] data lake at scale. Uh a bit about my background before I made terrible terrible financial decisions to become a founder. I was working at HSBC, one of
[00:58] the world's global banks, and we were building out their cyber data platform entirely on Databricks. We proved incredible value in terms of detection, throughput, accuracy, and cost.
[01:10] However, I must tell you that if you decide to build a cybersecurity platform in a regulated industry, you are possibly mad. And before I was doing that, I was working at KPMG as a delivery manager for cloud.
[01:22] Now, why do we want to do cybersecurity on Databricks at all? Well, it's very simple. Cybersecurity is truly a data problem. It sucks, ladies and gentlemen. If you
[01:34] have ever tried to work with cybersecurity data, you will have experienced several magnitude orders of pain. First of all, you've got the problem of volume, the sheer amount of data flowing
[01:46] through your lake. Now, of course, if you've got a small estate, perhaps one file, maybe it's manageable. But once you get to the size of a global bank, you've got terabytes per day of firewall, proxy, endpoint, etc. And
[02:01] what's worse is that all of these data sources are outputs from shall we say interestingly created software. Um we are not consuming nice, clean reference data such as you would
[02:14] get from a database. We are taking the digital exhaust of your estate. And that leads to the second problem of diversity. We are dealing with hundreds, sometimes thousands of different data
[02:26] sources. Different vendors, but also within each vendor you have the different schemas because when a vendor changes their output, they don't tell their cybersecurity team. And that's when you start to get into
[02:39] the foundational problem of data quality. We are not simply dealing with the problem of types. Is this a string or an int? We are dealing with the sheer problem of the fact that raw data is missing because people use something like UDP syslog, which is lossy.
[02:52] So, why does Databricks solve this and how can we do some real benefits in Databricks here? And how can Ziggur help you with that? Well, the first challenge is that data collection.
[03:05] So, at Ziggur, what we do is we manage the collection infrastructure for your estate. That's not just simply making API calls. There's often ancillary parts to data collection that are not purely streams.
[03:19] You've got Kafka streams, you've got event hubs. You might have S3 buckets, SQS queues. Now, in your current estates today, you may well be managing this third-party infrastructure either yourselves directly, in which case you have to learn how to use the cloud
[03:31] account, or you rely on a third-party team and have to go through some form of change process or governance. This adds friction, this adds delay. More importantly, this adds confusion. When you go to that AWS account and see
[03:44] there's 15 different S3 buckets, which of them are they related? How do they relate to the pipeline? That's what Ziggur Courier is meant to solve because we handle all of the collection, not just the easy bits, and
[03:57] we maintain the health and the pipeline and the back pressure and the resilience of these data pipelines. And if you have managed any of these yourselves at any meaningful scale, you will understand why that is a workload you want to get
[04:09] rid of. Now, the other problem with data I already brought up was the volume. Now, the fact that Zigger's Vault essentially gives you the ability to retain all of this cyber data in Databricks is very straightforward. We all know that
[04:21] Parquet is a fantastic format. Delta gives us so many benefits including time travel, high compression. Traditional security archiving tools put this data into an S3 bucket where it's incredibly difficult to use. So, having
[04:34] it in Databricks and accessible is awesome. And finally, we have the problem of diversity. The data management plane of cybersecurity is incredibly painful because there are no cybersecurity data
[04:48] engineers. That's because cybersecurity is often deemed to be a cost center. That means they have struggles on their budgets. No one's going to send their latest and greatest trained Databricks engineer to
[05:00] work on security. They send to the business line. And that's why some of the area that has the most difficult data transformations has to work on their own without support. And so, that's where Zigger's fits in.
[05:14] Um we will sort that for you. We will sort the collection. We will sort the context.
[05:28] So, here we have our zero-to-hero slide. We'll get the data now. We'll put it in open format. And we'll keep it for you. And when that data is ready, when that data is enriched, when that data is filtered, we will send it on to your
[05:40] seam, your operations tool. So, if your operators today insist that they don't want to leave Splunk, that's fine. But let's not just put everything into Splunk. Let's clean out the things we don't need.
[05:52] And as you can see in the architecture diagram here, Zigi's acts as a global filter. The data comes in, we send most of the data off to cold storage, and the stuff we need goes off to the index ready to be consumed.
[06:06] We are at Zigi's a built-on Databricks partner. That means there is no magical platform where the data has a second copy. What we are doing is giving you the managed
[06:18] Databricks experience that you want, but without the effort of having to get from A to B. So, I'm going to give you a quick demo of the product now, because of course, that's why I'm here.
[06:35] So, this is a visualization, he says. There we go. That is not working. There we go. Excellent. That's not displaying anything. Marvelous. There we go. We shall deal with these IT problems,
[06:48] ladies and gentlemen, and I shall drive this screen like this, and it shall be marvelous. So, here in a slightly overblown form is the picture of our semantic layer. So, everything on the left here represents the raw data as it comes in.
[07:02] And this is a demo environment, but as you can see, even with a small environment, we're already dealing with multiple different sources, and the assumption is that most of these may be at least a terabyte a day, sometimes more.
[07:14] Each one of these little boxes represents not just a vendor's data. It is the transport. It is the schema. It is the version of that schema. This is the data engineering context that is required to turn bytes into a
[07:28] table. And we maintain that parser, that config, that configuration for you, and we handle it as part of our collection and ingest. This is a critical point, because if
[07:40] your data structure upstream changes, you as the data engineer today are responsible for making sure that pipeline flows. And as I said before, the cybersecurity team are probably already underfunded in that department. So, whilst that data engineering is not
[07:54] completed, we are at risk. We lose visibility. That can be critical. That's why we have our semantic abstracts on the next layer. Because whilst a CSV may have a change in column
[08:07] three to four, or a JSON file may add or remove keys, what a vendor means does not change. The semantic understanding of unauthorized, unauthenticated, source, destination. And that's how we
[08:21] handle schema drift for you, not just as a data engineering construct, but also as a cybersecurity one. And the reason we can do that is myself and my team have been doing this for
[08:33] years, and we know how the data engineering needs to work for you. Because whilst your organization may be special, your CrowdStrike data is not. It is a commodity. Everyone has the
[08:45] CrowdStrike, the Entra logs, the Palo Alto flows. And at the moment, every one of you are writing your own data pipelines, and some of them are fantastic, and some of them are suboptimal. By consolidating that data engineering
[08:57] into a managed service, that means you are not spending your precious time on commodity engineering. You are focusing on delivering the value, understanding your security use cases. And where we get to the cool bit is our
[09:10] semantic layer. So, you'll see here that there are lovely green lines. They're going from the three different firewalls in the state. We've got the Check Point, the Palo, the Cisco. Because when I'm writing a security
[09:23] detection, do I care which vendor specifically did the did the detection occur in? No, I do not. And we did some analysis on lots of
[09:35] different data models, such as the common information model, and lots of open source uh YAML security searches. And what we found was very interesting, ladies and gentlemen, 80% of the data in a security event is never used for a
[09:50] detection or threat intel. So, if you've got a table or a JSON payload that has 50 keys, you are using maybe four of them.
[10:02] And in the traditional SIM architecture, we send all of these columns, all of these data payloads, regardless of their relevance to a tool, which charges us handsomely for per gigabyte ingest.
[10:14] This needs to change. This is uneconomical, this is impractical, and it's filling ourselves with irrelevant data. And the modern world is getting more complex. We are not lessening our data volumes, we are increasing it. We're not
[10:27] reducing our number of activities, we are increasing them. We have more and more non-human identities performing activities that previously would have been restricted to a handful of users. And so, our security context become ever
[10:39] more complex. That's why the ability to build your detections, your payloads, your dashboards on a semantic layer reduces that cognitive overload.
[10:51] Today, if you have three sources of firewall information, you have two choices. You write three different queries in whichever detection tool you choose, your Sentinel, your Splunk, your
[11:03] SentinelOne. And that means you have three queries, three alerts, three sets of things to change when something goes wrong. Or you write one search. But that search has to include the data
[11:15] engineering task of converting what does SentinelOne mean for this particular event, or what does Palo mean versus Cisco? That is what our semantic layer highlights for you. Is that there are
[11:28] multiple stages of the data ingestion process. And for those of you who are expert data engineers, you may think, of course, that's obvious. How did they not see that? But you got to remember that the area
[11:40] we're working with is not one which has a traditional background in data engineering and data science. And moreover, needs the assistance and need this made clear for them. So you go through the layers, you go through the models, and these semantic
[11:53] models are originally maintained and looked after by us, Zigers. But they are fully extensible. And that's where the fourth presentation layer appears, because every business
[12:06] will have their own unique characteristics. And it would be hubris to suggest we could manage all of them. But what we provide to you is the tooling and the extensibility and the clean baseline for you to begin your
[12:18] detection engineering. We also allow you to do detections because we do strain stream processing as part of our ingestion process. So rather than wait forcing you to create
[12:31] your own streaming tables up front, if you wish or have an existing Sigma rule or Sigma search which will determine a detection, we can clean that up up front. And that means what you end up doing is you create smaller, cleaner, thinner
[12:46] data sets which are enriched. And the reason I can perform enrichment, of course, ladies and gentlemen, is I am built on Databricks. And when I say that, I mean that explicitly. The product you get is not
[12:59] just this very pretty GUI which has cool data engineering and tool graphing, but you get the actual benefit of a fully managed Databricks workspace.
[13:11] Whoop, there we go. Click the button. With all the full fat Databricks features. Every Databricks release that you hear today will be available to you. We are not abstracting it, we're not hiding it from you.
[13:24] What I am doing is preventing you from making mistakes. So, the data catalog that I provide has a strict naming convention, and you cannot bypass it because you do not put
[13:37] the data into the lake. I do. You cannot incorrectly configure the lineage because you are not controlling the lineage. I am. You cannot forget to fill in your Unity catalog definitions because you were
[13:49] bored, or busy, or forgot because I am doing it programmatically. And this is the real use case once you start to go beyond simple scale problems.
[14:02] When you've got a couple of sources, or a couple of gigabytes per day, that kind of management is is fine. But when you're talking proper enterprise scale with hundreds of workspaces, hundreds of data sources,
[14:14] you will end up spending more time on the churn of just getting data in healthily, on time, without terrible latency, and then the actual use cases, the
[14:27] business value takes longer and longer to achieve. And that's where a platform like this can really accelerate you because you all have a finite amount of time and resources. So, you should deploy them in the
[14:39] locations where it's most useful. And because my data is clean and in Unity catalog, all of the lovely new features work perfectly. If you ask anyone today, "How does my
[14:52] Genie workspace do perform?" Some will say it works fantastically because they've got some good engineering. Some will say it gets a bit confused. And you say, "Have you given it context? Have you given it that information?" And then we
[15:05] have to take them on that journey. What we are doing here at Zig is is ensuring that the journey is already provided for them. They already have the support. They have the tooling that they need.
[15:18] And as I said, we handle the collection at both the infrastructure and the pipeline and the hygiene layer. So, we will give you a suite of connectors. And those will include running cloud
[15:31] infrastructure on your behalf within our cloud accounts. Why is that helpful? Well, if you look today at the concept of having to ask someone else to run an S3 bucket for you, it means you don't have to worry about change management, compliance, key
[15:45] handling. I will do it for you. And the reason I can say that with confidence is because I have built multiple cyber data lakes for multiple highly regulated customers.
[15:57] The reason myself and my colleagues set up this business is because we went to multiple banks, governments, retail giants, and we saw the same problems time and time again.
[16:09] The well-meaning, well-intentioned people were fundamentally swamped and building those baseline data engineering tasks just takes too much time.
[16:24] And when you want to investigate or review your data, we also control the ability to say, "You know what? You don't need to investigate everything." So, rather than giving you read access to the full lake, which we can provide,
[16:36] if you've got many departments or use cases, we allow you to create subset environments and we will handle the R back, the access control, the monitoring. And we're not double copying in this case. We're simply using the ability of
[16:49] Delta Sharing. Similarly, the Databricks workspace that we provide to you is run in my workspace on your behalf, but if you wish to use it in your own, we simply Delta Share it to you. So, it becomes like a managed platform
[17:02] ready workspace. And I will now boldly try and press this button, which is not going to work, and get this PowerPoint to the final
[17:14] slide, which is a very risky and very dangerous idea. There we go. It worked. Who knew I've got over a decade in IT?
[17:27] So, the recap. We take it and at scale, we manage the complexity, we manage the load, we manage the resilience.
[17:41] The point is not that you could not do it yourself. The point is that if you're doing that yourself, you are not doing something else. We are doing the basics and we are doing them well, and we are doing them
[17:54] repeatedly, so you can accelerate your cybersecurity journey. And of course, every demonstration has to end with a pontificating quote. Everything war in war is very simple,
[18:07] but the simplest thing is very difficult. Because every CISO will tell you, "Can't you just put it into this data brick tool and tell GD to answer it?" Well, I could.
[18:20] But the effort of getting from that individual single terabyte stream is the effort I don't want to face. Thank you very much, ladies and gentlemen. Uh E, please complete your survey and say nice things about me
[18:33] or terrible things if you didn't like me. And most importantly, please find me on LinkedIn. Happy to have a chat after the session. Thank you very much for your attendance on this last afternoon. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.