Atlassian Cybersecurity Data Standardization with OCSF and Databricks

Summary

Atlassian standardized fragmented security logs from 300+ million customers and two decades of acquisitions onto the Open Cybersecurity Schema Framework (OCSF), normalizing 44 billion events per day.
An ETL-first design with type-safe Go structs and an agentic mapping workflow standardized 500+ security data sources while keeping the architecture open and vendor-independent.
Databricks optimizations — Delta Lake, Z-ordering, partitioning, and Photon on Graviton — delivered major query-performance gains and enabled AI-assisted, risk-based threat detection.

Atlassian Cybersecurity Data Standardization with OCSF and Databricks

https://www.youtube.com/watch?v=75sDMxRXQ1g

In this video from Databricks, Zach Schmerber — a security and logging architect at Atlassian and a maintainer of the Open Cybersecurity Schema Framework (OCSF) — explains how Atlassian tamed years of inconsistent security logging. By standardizing on OCSF and building on the Databricks Data and AI platform, Atlassian normalizes roughly 44 billion security events per day into a queryable, governed security lakehouse that powers 500+ production threat-detection signals.

Chapters

What is OCSF and why did Atlassian adopt it?

The Open Cybersecurity Schema Framework is an open, vendor-neutral, strongly typed schema for security events. Atlassian chose OCSF to unify the schemas of many different security logs, improve event correlation, avoid vendor lock-in, and accelerate detection development. An open standard meant the team could normalize data once and reuse it across tools rather than rebuilding mappings for every vendor.

How does Atlassian standardize 44 billion events per day?

Atlassian adopted an ETL-first architecture: data is normalized to OCSF as it is ingested, rather than translated at query time. To keep transformations correct at scale, the team embeds OCSF directly into native Go structs, which gives them compile-time (type-safe) validation of every mapping. More than 500 mappings are automated through an agentic workflow, and every change is regression-tested before it reaches production.

How is the lakehouse optimized on Databricks?

The standardized events land in Delta Lake and are tuned for fast, cost-efficient investigation. Atlassian relies on partitioning, Z-ordering, and Photon on Graviton, along with Spark autoscaling, to deliver large query-performance improvements. On top of the standardized data, the team built an Alert Explorer plus signal- and risk-based detection, and is using OCSF's consistent semantic context to enable AI-assisted security analysis.

FAQs

What is the Open Cybersecurity Schema Framework (OCSF)?

OCSF is an open, vendor-agnostic, strongly typed schema for normalizing security event data. It lets teams map logs from many sources into a single consistent structure so events can be correlated and queried uniformly.

Why did Atlassian choose an ETL-first architecture over schema-on-read?

Normalizing to OCSF at ingestion produces clean, consistent data that supports real-time detection and fast queries, instead of paying repeated translation and correlation costs at query time.

How does Atlassian keep 500+ OCSF mappings accurate?

Mappings are embedded in native Go structs for compile-time type safety, automated through an agentic mapping workflow, and regression-tested before deployment so schema changes don't silently break detections.

What Databricks features drive the performance gains?

Delta Lake storage, partitioning, Z-ordering, and Photon running on Graviton — combined with Spark autoscaling — deliver substantial query-performance and cost improvements for security investigations.

How does OCSF support AI-assisted security analysis?

Because OCSF gives every event consistent fields and semantic context, AI and agentic tools can reason over standardized data — enabling capabilities like conversational incident response and contextual, risk-based threat detection.

Full transcript

[00:08] All right, welcome everyone. My name is Taylor Cain. I lead marketing here at Databricks for all of our cybersecurity business. Really excited today to have Zach from Atlassian. Atlassian actually won our transformation award last night at our Databricks customer awards. So today you're going to hear from Zach
[00:23] on their use case, how they're using the Databricks platform to do an awesome cybersecurity use case. So with that, Zach Schmerber, could you come out? >> Everyone. Thanks. Thank you.
[00:40] Excited to show you what we've been building. It's been a really fun journey over the last 3 years. And a lot's changed in the last 6 months. So this is our journey into data data standardization. And a little about me. I'm Zach Schmerber. I
[00:57] have been driving sort of the standardization of logs at Atlassian for the last 3 years, and I've been also working heavily in the OCSF, where I'm a maintainer of the schema, along with all of my colleagues. I'm really excited to show you what we've
[01:12] built, and all the exciting things we'll see in the future by using the schema. I spent the last 15 years as a cybersecurity and logging architect, essentially, all the way back from my Splunk days when the cool new thing was just combining a
[01:28] few different fields to petabyte scale data architecture here at Atlassian. So the background and prerequisites a little you should know about this before we even dig into what we did is what the
[01:43] OCSF is. And it's a unified framework that allows teams to normalize data across huge swaths of of different chunks of data into a standard look a central
[01:59] location like data bricks. It also allows you to use the data outside of data bricks if you choose. And the beauty of it is it's all the same framework. And ultimately the goal of it is to shift it left all the way to the emitting entity.
[02:14] And it has open governance. So the Linux Foundation supports the OCSF and ITU should be shortly supporting it as well. We are working with all the federations in ITU to try to make sure that this
[02:31] standard becomes an international standard and we can all start with the same canonical data set to work with which will make cyber security a little bit easier because it's already hard enough, right? So I made a video. I got to hit play here.
[02:51] And that way I can So while I go through this, the other side can take a water break. And the thing we're going to try to focus on here is understand what the schema is so you have a little bit of pre-requisite information about this talk. The OCSF has been around for about
[03:09] three or four years now. I started all the way back in RC2 to >> [snorts] >> adopt the schema and understand it. Back when we could manually transform data from its raw format into the OCSF. And boy things have changed. We'll go into that in more detail
[03:27] in the rest of the presentation. But you can see in RC2 it was quite a bit smaller and we've since expanded many different classes and profiles to extend the schema to capture data
[03:42] from anything from identity and access to unmanaged real systems. So it's expanded quite a bit. On the left we have extensions and these extensions are specific to operating systems. So, when you have one selected, or you data that is specific to that operating
[03:58] system, it's usually not found in what else. On the left, we also have profiles. These profiles um append additional information to the existing classes. So, if you have host information uh that you want to add to a class, you
[04:13] can check that, and previously the actor device fields did not exist in that class, but I can check them in, they will. Uh the schema itself is represented in JSON. So, if we wanted to look at a particular class, and see what
[04:29] it looked like in its JSON representation, we could click here. And that's no fun to look at, so we use schema browser. Um in the end, this whole rep this whole um schema browser is just a way to pivot through the data. You can compile the
[04:46] schema into a useful format for schema validation, and data bricks later, uh and there's many details on how to do that, but instead of telling you how to do that on this, I'd like to say thank you for watching, and uh go ahead and Zack after you're done drinking your water.
[05:02] >> [snorts] >> All right. So, now that you know what the schema is, and what we're working with, and building on top of, uh let's get into the problem that we had at Atlassian. And before I go any further, uh are there any questions? This is a small group, so I think we can
[05:18] do question and answer. Not doesn't look like it. So, let's get into the problem. So, the enterprise challenge we had is a global um SaaS provider had 20 years of build up to 300 million plus customers, and
[05:34] rapid expansion. So, you can imagine what that does to our logging pipeline. We had data inconsistency, multiple companies we purchased with different logging formats. We had to coalesce that data into a central location to make any sense of it. And it's coming at you at 44 billion events per day. So, what do
[05:49] you do? Uh you got varied timestamps, you got proprietary schemas, you've got operational drain for anyone trying to organize and structure this information. You've got analysts wasting time on normalization in the pipe after the data's been landed. You've got limited
[06:06] cross-platform correlation because of that, and maybe the data's not even stored in the same place. You've got strategic barriers with SIM pipelines breaking down at scale, and vendor lock-in with security tools that won't even show you what you want to see anyway.
[06:21] So, how do we solve the problem? >> [clears throat] >> Strongly typed schema was critical, and this is why we chose the OCSF. Uh there weren't a ton of options out there. There's ECS, there's UDM, there's a few others. But this is the only one that was considered wholly open.
[06:38] Uh it ensured a strong, robust validation of all of the data that we were transforming, and minimized the ambiguity within our data sets after they were transformed. Compression efficiency was good, and that I'll get to why that matters for us. You may not think it matters because
[06:54] we're talking about Databricks for the most part, but it does. Uh it's extensible, and uh it was open source. So, they had a community where if I needed to make a change or enable something different in the schema, I was able to do that. Uh investigations where uh single query
[07:12] could do many different data sources because of the data structured. Maybe you've got five vulnerability scanners in your system, not uncommon for large orgs. Now you have one data set to deal with instead of five. Uh semantic context remains with OCSF. A lot of people look at it and go,
[07:29] "Wow, it's really big. There's 40,000 different uh fields that I need for this particular category." And yes, it is, uh but we'll also get into why that's valuable in the future here.
[07:47] So, at Atlassian, um we do ETL and not ELT. So, do I have anyone going to go boo? We don't We don't We don't like that. No? All right. Well, there's a reason for it, and I'm excited to tell you about it. Uh real-time detection capability has been enabled uh by doing this. Consistency and vendor independence. So,
[08:03] if we need to switch a vendor, we can. It keeps us in control. Operational efficiency of being able to put in PRs to our custom ETL tooling uh has allowed us to make a really fun transformation engine where we've been able to assist the customer agentically
[08:20] and speed up the biggest bottleneck for us, which was our data transformation itself. >> [snorts] >> So, uh got any SQL people who believe that is the best language for transforming data and mutating it?
[08:37] All right. Well, I'm here to argue with you later after this is over to talk about how type Go structs has enabled our data transformation process and given us the ability to automate the most difficult part of transforming data.
[08:53] So, the power of type safety was really critical in this in the sense that we were able to validate at compile time. We had nested objects that are part of native Go structs. And I guess I should explain this a little bit more, but the Go structs, we embedded the entire OCSF schema into Go structs.
[09:11] And what does that mean? That means every time you produce a a new OCSF log, it's pre-validated because it can't be anything else but OCSF. Those are the only structures that exist. Uh the deployment because of that can be anywhere, and it's unit tested.
[09:28] So, in the agentic feedback loop, this reduced our token usage pretty heavily when we automated the transformation process because you're not having a hallucination cause you're multiple iterations of mappings through the Agented Flow.
[09:44] Uh platform selection was a critical decision factor for us. Uh it was pretty obvious which one we chose. Right? I'm here talking to you. So, uh Databricks was the winner because we already had it. One, two, it helped with native support with asset transactions, made it easy to move through the
[09:59] different versions of the schema without breaking a bunch of things. We still break things, don't get me wrong. It's just easier to to work with. Um auto scaling of Spark clusters is a big win considering we aren't the only people using the clusters and collaborative informative security
[10:16] teams. Sorry, I got some feedback on that. Uh managing infrastructures uh and overhead was was easier with a platform like this. It's what it was designed to do, right? It's designed for large corporations that have many users and and that's why we used it.
[10:33] Table optimization was super critical. Um I mean, this is basic stuff, but if you partition something, it reads faster and see you get improvements in your query ability. We saw something like uh with our structured OCSF data, 45
[10:48] seconds on a query, and after optimization and partitioning, two to three seconds. So, it was a 95% improvement. Same thing with our larger queries, just about 94%. >> [snorts] >> So, the part I'm really excited to talk to you about is the Agented OCSF mapping
[11:03] workflow that we developed. Initially, um this was developed for the data engineers, me and and and my team, the Megabees is what we call ourselves. And it was a six-phase procedure where we would collect logs, uh transform them into
[11:20] Type Go, route them to their destination, test it, configure, and write a PR, and then push it up. Uh over time, we realized that most of this is can be most of this can be automated, and eventually it became so automated that we passed it back to the customer. So, we no longer have to do data
[11:36] engineering on the data engineering team when it comes to that messy cleaning the log stuff that no one likes to do. >> [snorts] >> In detail, the log analysis is done by the analyst. They determine which logs they want to use. They give us a canonical data set that they want transformed. Make sure it
[11:51] has every eventuality of what that log would have inside of it. Then, after that, a gated confirmation, um will allow them to the next step, which it does a scan of the framework that's inside the ghost trucks itself. That's That's That framework scan then
[12:07] informs the type to go generation of the mappings and the DAG. So, you can give me a all the cloud trail logs, and it will itemize and discern and discern which logs should go to which part of the schema, and then generate the mapping for it. So, it creates the
[12:22] mapping, the DAG, and finally, it routes and tests those mappings locally, validating that they come through as expected. After that, it deploys a canary config of a PR that will go into the canary environment and validate at scale that all these different mappings will
[12:38] actually work in production. At which point it sat there for a day or two, it will deploy itself to prod, and notify the end user if there's any errors. So, you can see how this would streamline the process of standardizing your data into a common model, or I
[12:54] shouldn't say model, a common schema. We'll talk about model and schema separation later after this, at the end. >> [snorts] >> So, what powers it? Uh the tool chain, we have an IDE that uh does schema mapping, and we have a Go assist of
[13:11] library, which I've put into a QR code that you can find uh at the end of this talk. Just pull out your phone, you should be able to link to it. Um there's some really fun performance enhancements for the validation process inside the ghost trucks library that allows us to take structures and validate them,
[13:27] Uh do object mapping and enhancements like unmapped fields is part of the OCSF. It automatically produces those and puts them in the right place. Uh schema helpers like IP path normalization are built into the go library as well.
[13:43] Um intelligent routing is is also part of it. So, uh with the helper functions and all the nice to haves, that's what gave us the capability to reduce the toil of mapping data.
[14:00] It's correct and safe and repeatable by design. We kind of talked about that already in a sense that the schema correctness is guaranteed with the go structs. Uh the data is auto derived and things like observables are automatically placed into the mappings themselves. Previously, something you had to either
[14:16] do manually or program carefully. Uh the regression testing has been super useful in a sense that we validate everything before we push it into production, which made me feel a lot safer about, you know, handing this work off.
[14:32] Uh the low onboarding overhead was probably the biggest winner of all this in a sense that I can now have a SOC analyst do principal data engineer work in a production environment and let it rip. That's pretty good feeling, right? And
[14:48] of course, we do check everything they do, but it it is nice to be able to have that possibility. >> [snorts] >> So, why OCSF and AI? Um what I really am excited about is what's coming and
[15:03] the pattern recognition that uh we can get with a fully um semantically accurate schema is exciting. Uh it enables accurate context aware schema mappings uh with the pattern
[15:19] recognition and the semantic integrity of the data is kept because of the verbosity of the schema itself. If anyone's ever looked at the schema, it's it it's hard to read as a human being, but initially what I thought was going to be the failure point of the schema itself turns out to be its
[15:34] greatest selling point in that the context that you need for an LLM to do assisted queryability inside of a data set now exists within the schema itself. And the logical continuity keeps
[15:49] pretty much intact during the AI mapping as well. Um, which preserves the links between fields and make sure that you have a mapping of what happened when this automated mapping took place. Quantifiable results that we saw with analyst productivity was up by 60%
[16:06] because instead of taking half their time uh, reviewing and validating that the field they're looking at was what is what is to be expected for a a single tool, they can look at all tools and know that they have a repeatable process
[16:21] to look through the data that they have. Um, having a quality data foundation allows for automation. And that means that you can automate your way out of those tedious tasks of um, trying to fix a simple problem in in a in a sock
[16:39] when it's really uh, repeatable after your data's structured. The storage optimization was improved pretty dramatically in the sense that now that we know where our data is, we know what we can draw down, what we can keep every time. You don't have to just keep all the data. Uh, you could reduce
[16:55] it, structure it. And on the wire, we are able to reduce it pretty heavily because we were able to compress the data as it moved through our pipeline. Not we're ETL, remember, so we have to transit this data.
[17:13] The observed benefits, uh, multi-domain correlation has been super helpful. The entity resolution within that, um, is really strong. Uh, keeping the semantic entity integrity in the OCSF has allowed us to look at user behavioral analytics in a really efficient way. Um, the pattern analysis and
[17:30] cross-application, um, analysis it is really nice. Let's say you've got five different routers and they all speak a different language. Well, now we can just use one language and write simple queries to identify anomalous behavior.
[17:49] So, this is actually Jordan, my colleague, who's going to give you a little insight into something we built custom for our signals, which is how we write a lot of our detections. We generate signals and then correlate them together. >> Hey everyone, I'm Jordan and I'm part of the Atlassian detection and response team. Adopting the ACSF schema has
[18:05] enabled us to transform our threat detection pipeline by moving from one-to-one alerts to a signal- and risk-based detection methodology that allows for multiple signal findings to be clustered together in a single alert. Here's an example of what an alert looks like for us today using a custom tool we've built called Alert Explorer.
[18:23] The alert contains a cluster of OCSF detection findings based on the OCSF username value, J. Doe. Instead of having to manually map different data sources within queries to a consistent schema or field name, we can know with confidence where different fields appear in the schema, which
[18:38] allows us to standardize clustering, enrichment, and triage plays. For example, instead of having per detection enrichment of user information, we can call an enrichment API every time we get a username value. Having standardized field names also
[18:55] makes it easier to pivot on specific values, or to follow standardized triage plays instead of using custom per detection response steps.
[19:10] Overall, OCSF has made it much easier to write detections and provide more meaningful and actionable alerts, which analysts can dive deep into regardless of where the data came from.
[19:25] >> All right. So, the key success factors we success factors we've had were table optimization and for query performance. The ETL first architecture was probably the biggest time saver in a sense that we've automated that process and what to to give you perspective on this, took me
[19:41] a year to map about 1/3 of the data that we have mapped today and 2 months to map the other 2/3. So, the acceleration on the transformation has really made the biggest difference. And as you well know, the more data you have transformed, the better your correlation
[19:57] and the deeper your signals become. The enriched data enables us to create new signal generation opportunities and the strict schema and Z-ordering enhancement of query performance. If you look on the right, you'll see the Graviton cluster with Photon. That giant cliff is actually dollar signs stacked
[20:13] up. And as soon as we turned on Graviton with Photon, you can see we had about a 2/3 improvement in the performance of our queries. So, lessons learned. Schema evolution is hard even with asset tables. Um
[20:29] when you when you do schema evolution, you need to a lot take in a lot of considerations of what your your teams are doing on the other end. And deprecated fields can be dangerous. And so, just be aware of that. Partitioning anti-patterns, don't over partition, obviously, right? Don't do
[20:45] don't don't make a thousand partitions when you need one. Um the underestimated hurdles were sort of the maintenance of the mappings themselves and figuring out a way that everyone using the schema knew how to modify them so that they don't break things and that
[21:01] in-stream enrichments were not affected by that. Uh also, you you we're continuously building tools to read and enrich this data as it goes through the pipeline. Redis infrastructure was an interesting one to deal with. Um the native Go structs from the start
[21:17] would have been really nice because we used a bespoke DSL before that with a lot of hallucinations and didn't give us the ability to quickly iterate through this transformation process. Uh I guess I'm kind of repeating myself here. The ETL platform were was part of
[21:33] the key takeaway. Um shifting from mapping as uh shifting from mapping as a service to self-service platform was huge. It can't I I I said that like three times now, but that's because it really matters. Um the validation automation with the rigid
[21:48] Go-based schema, uh that was also huge in the sense that you didn't have to worry about waking up in the middle of night why you broke something or why your PR crushed the system or how why is there you know, 4 billion errors today in our environment? Cuz
[22:04] it it it it was just a piece of mind as a data engineer to not have to worry about jumping on an on-call because you you pushed a PR on a Thursday and it broke everything on Friday. Um so optimizing the table structure, understand your I think this is a key one that that is based on each different
[22:21] organization. If you know your data volumes, you know your customer uh you can structure your tables to fit that. Um you know, there's different ways to optimize the ORC SSF for different scales. If you're Apple you're probably not going to want a
[22:36] single table with a ton of partitioning cuz you've got three petabytes a day coming in, right? If you're a small startup, maybe you put everything in one table and partition it that way and keep it simple. But in the end, try to simplify it as much as possible for the end user while
[22:52] still keeping performance. That's probably obvious. Now, looking ahead I'm going to take a pause cuz we we went through a lot. Is there any questions about what we did before I talk about what I'm excited about in the future?
[23:16] Uh we've got a mic headed your way in just a second here.
[23:39] I have such an echo up here. I'm going to tell you what.
[23:56] We consolidated every log we have for security into the OCSF standard and then dropped it into its location. So, we have over we have over uh 500 different mappings and because we've automated that process, it's no longer a slog to try to get them all to join together.
[24:22] Yeah, they can they can use what they we standard we were able to standardize our dashboards because if all the logs have the same objects, the schema shares objects across different classes. So, therefore an actor.user in authentication is the same actor.user that's going to be in a network event.
[24:39] And so, now our dashboard our dashboarding has been simplified dramatically and you don't need, you know, the IP.X-Forwarded-For and IP address and all these separate ones. You've got them standardized. Yeah.
[24:57] Any anything else before we go on to the fun stuff? More probably more if you depends on what you consider a data source. How many data sources are in CloudTrail? Um is that one data source or is that
[25:13] is that 1,600 cuz there's 1,600 different logs in there? Uh I guess it depends on how we classify it. We have Okay, if the the go structs are fairly simple mapping and we have 40,000 lines of mapping code.
[25:29] So I don't know how to quantify that. >> [laughter] >> Uh So the opportunity that I want to talk about is what's coming next. Once you have created a data foundation
[25:44] um the agentic assistance in the querying of this data is where it's going to be fun. Uh you can I think the big quotable quote talk to your data, right? How do you talk to your data unless your agent has context?
[25:59] It it's it'll hallucinate and give you the wrong answer when you talk to it. It's like saying orange and then I want a full response cuz it doesn't have the context of what I'm talking about with that orange. The beauty of the OCSF is the fact that it has the context built into the schema with the definitions inside of it.
[26:16] And that agentic assistance will now be more accurate with the schema. Uh the skills training uh I'm excited about being able to train skills to understand you know, what they're using in the
[26:32] schema on Can I take my metrics from my vulnerability analyst, look at the fields they're looking at all the time and then feed those fields for context into uh the skill itself to make it [snorts]
[26:47] repeatable. Uh that means that I'll have eight an orchestration of agents that will be my vulnerability analyst doing threat hunts on vulnerabilities for me with the context necessary to be accurate. And uh contextual sequel. Like uh you've
[27:05] you've got a question, where is my DNS C2 traffic in the scheme? I've never done cybersecurity. Tell me what Tell me where it is. Theoretically, because we have all the context in the schema, and you're able to easily find it, the agent should be able to tell you the
[27:20] query you would need to achieve this. So, simplifying these complex processes, codifying what's in my head and putting it into a skill. That's what's exciting. And the advantage of the OCSF means we
[27:36] can also work together to build context graphs. This is something I'm going to be working on pretty heavily, and I've got a couple of QR codes after this, which will give you a link to the OCSF so you can join us. Uh it'll also give you some of the repo open source repos we use to build the agentic flow.
[27:55] >> [clears throat] >> Oh, shoot. Too far. Um semantic mapping, uh I really want to get to the point where the OCSF is the canonical starting point. It's it's a schema, and it can start everything there. A lot of these models we talk about with
[28:11] ECS, um what's some other popular ones? UDM. They're they're they're coalescing existing data into another shape, which makes it a model. So, even though I call it a lastic common schema, not a lot of things Some things emit it, but not everything emits
[28:26] elastic common schema, and therefore it is mostly a model. What I want to get to is where everyone emits OCSF, and that's the canonical starting point where we can build a semantic model on top of the data coming out of it. And community success. So,
[28:43] the best engineers I've met have been in the OCSF community. Um the things I've learned that I'm talking to you about today were mostly learned in that community. And even more powerful than a schema is the people that work with it
[28:58] and that are excited to advance it. So, I hope you can join us. The schema browser is found in this QR code here. The go OCSF repository that we'll be updating shortly with all the helper functions that we did for the ETL engine can be found on this other one.
[29:15] And while I leave that up for a second, I'm going to take a sip of water and see if there's any other questions. Then we're going to go to one more slide with my LinkedIn and some other interesting repositories for joining the OCSF.

Learn more about building a security lakehouse on the Databricks Data and AI platform, and explore the Open Cybersecurity Schema Framework project.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.