Safe Skies: Data Contracts as Code with Databricks, DQX, and Ontos

Summary

Boeing operationalized data contracts at enterprise scale using the Open Data Contract Standard on Databricks, replacing manual data stewardship across hundreds of federated data products with machine-readable governance as code.
The open-source toolchain combines DQX to enforce data quality rules at the pipeline level with quarantine and drift alerting, Antos for contract lifecycle management and discovery, and GenAI to auto-generate contracts by inspecting data profiles.
Contracts are operational systems — not static documents — that notify subscribers on schema drift, enforce compliance through automated queries, and make data quality a preventive control rather than a gate check.

Safe Skies: Data Contracts as Code with Databricks, DQX, and Ontos

At Boeing, bad data is an operational risk. A data quality issue in predictive maintenance can corrupt telemetry, delay flights, and strand 300 passengers. To scale governance across hundreds of data products and federated teams, Boeing replaced manual stewardship with governance as code: data contracts as machine-readable ODCS (Open Data Contract Standard) specifications, enforced by DQX, and managed via Antos.
Learn how to operationalize data contracts across enterprise scale. Covered: standardizing on ODCS and ODPS specs (vendor-neutral, Linux Foundation maintained), using GenAI to auto-generate contracts by inspecting data profiles, integrating DQX to enforce quality at the pipeline (not the gate), quarantine bad data, and alert on drift, and using Antos for contract lifecycle, discovery, and compliance as code. This field report includes live demos of contract authoring, DQX integration, quality enforcement, and schema drift detection.
🤝

Chapters

FAQs

What is the Open Data Contract Standard and why did Boeing adopt it?

The Open Data Contract Standard is a vendor-neutral, Linux Foundation-maintained specification for defining data contracts as machine-readable ODCS files. Boeing adopted it to standardize governance across hundreds of data products and federated teams, enabling automated enforcement rather than relying on manual stewardship.

How does DQX enforce data quality rules inside Databricks pipelines?

DQX integrates with Databricks pipelines to execute quality checks at the point of data processing rather than at a separate gate. It can quarantine records that fail quality rules and send drift alerts, ensuring bad data is caught and contained before it propagates downstream.

What role does GenAI play in Boeing's data contract authoring process?

Boeing uses GenAI to auto-generate data contract drafts by inspecting data profiles, significantly reducing the manual effort required to author contracts for new data products. Authors can then review and refine the generated contract rather than starting from scratch.

Why does Boeing treat data contracts as operational systems rather than documents?

A static document has no enforcement mechanism and quickly goes out of date. Boeing's data contracts are machine-readable specifications that actively enforce quality at the pipeline, manage schema drift detection, trigger subscriber notifications on changes, and support compliance queries — making governance a continuous operational process.

Full transcript

[00:08] All right, everyone. How many of you have experienced a flight delay and been stranded in airport? Raise your hands. Okay, I knew everyone was going to raise their hands. How about just for this conference? Who had that delay on Monday? Yeah. Not a great day for the airport. But, have you ever thought the cause of that delay, maybe it was a data
[00:23] quality issue. A miss on our one of our predictive algorithms for maintenance can cause with maybe corrupted telemetry data can cause a delayed flight and affect 300 passengers at the gate. But, don't worry. After this talk, I hope you'll have confidence that we have solved that problem and we've got you
[00:40] covered. But, bad data is still a huge operational risk for us. So, what we're going to talk about today in one word or two words going to be data contracts. But, for the next 40 minutes, what I really want to talk about is how a data contract is an operational system, not a
[00:55] static document. And we're going to show you what it takes to operate that across an enterprise and a federated scale. We'll start with the foundations. What is a data contract? How is it enforced? How do you author it? Life cycle, stuff like that. Then, we're going to go into
[01:11] implementation and tooling and the open source stack that we've adopted in Boeing. And throughout all this, we're going to treat this as a field report talking about lessons learned, give you the honest details, and we're going to try and do this through live demos. So, wish us luck there.
[01:26] Hey, Michael, tell us who you are. Oh, I missed the part. I missed the slide. Sorry, guys. Uh my name is Michael. I'm a solution architect at Boeing. I I work as a uh a platform um architect uh with the um data mesh governance team. And uh I know everyone here has probably
[01:43] heard of Boeing and you know that we manufacture huge planes and that's the main part of our business that we do. But, we also create software and solutions that help airlines operate those planes and that's the part of the business I work in. And this is my co-presenter, Alan. Hi, I'm Alan Reese and I am a Databricks
[01:59] uh forward deployed engineer, which means that I help our customers to put data solutions into production. And I've been working with the Boeing team, the Boeing account for a little over a year helping with the data mesh and with enterprise governance. Thanks.
[02:14] All right, so take a little step back just to talk about the reality of operating a data mesh at enterprise scale. We have hundreds of data products, hundreds of users, soon hundreds of agents across many domains and different business units. And what we learned is that manual stewardship in organization
[02:31] just can't keep up. And the blast radius has also grown. Failures now cascade and affect more and more teams. Back when things were smaller and centralized, you had this one one group that was kind of managing all your data engineering and we had this implicit trust of proximity. We knew who to go
[02:48] to, they knew all the context. But scale has forced us to decentralize. Now trust is trust is harder to establish and verify across all these boundaries. What we learned also is that data quality issues start to surface along
[03:03] these interfaces between our teams. And this scale problem is only growing as I know you're all aware. So with decentralization, we we solved the scale problem, we have autonomy, and domain expertise is now in the hands of teams producing the data. So that's all great.
[03:20] But that spreading out of this operation in this organization has led to some lost trust as I was saying. And you just can't assume that that trust is going to organically manifest across the mesh. So to to solve this problem, we needed guardrails. And we had to move away from
[03:36] gatekeeping. And we needed a contract to help formalize the interface between these different teams and producers and consumers of data. And especially as I alluded before too, we really needed to enforce quality, quality of the data across this the mesh.
[03:52] And we want all of this to be governed as code and automatable. So, throughout the talk we're going to hit five main topics. We're going to talk about standardization first. So, we know we need contracts, but but a contract by definition has to be agreed
[04:09] to by all parties. So, first you have to standardize. And the key decision and lesson learned here for us is don't invent your own, there's already an open standard. We write our standards in ODCS, the open data contract standard. It's a
[04:25] vendor neutral open spec maintained under the Linux Foundation. And there's a companion standard called the ODPS for data products. It's an open spec, it's portable, it's actually tool agnostic, and it outlives any one team.
[04:40] It's also importantly a machine-readable format, and you're going to see that later in our demo what a difference that makes when it comes to automation and agents. Next, we're going to talk about authoring. This is a huge area of friction across all our teams, asking them to put in this extra effort to create a a definition of their data,
[04:58] and we're going to show you how we make that a lot easier with AI. Then, we'll talk about enforcement. In enforcement, we're talking here about compliance and quality. And you want to enforce this at the pipeline, not at the gate Sorry, at the gate, at the pipeline, not at the
[05:13] consumer. And then we're going to highlight a little bit about discovery, how important that is, that you know, consumers and want to be consumers are able to find data across your mesh. And last, we'll talk about maintain. How do we build in compliance? What if a contract breaks?
[05:29] Every subscriber needs to know, and we'll show you how we did that at scale. I hope that helps with the context. Now, I'm going to pass over to my co-presenter Alan just to give us the high-level architecture. All right. So, thank you, Michael. Um so, this at a high level is the
[05:45] architecture that we're using to operationalize um the aspects that Michael just talked through, right? So, like anything in Databricks, at the foundation we start with Unity Catalog, right? I'm going to point out pieces of this architecture that are generalizable, that are open source and beyond just the
[06:03] Databricks stack. You can incorporate Databricks as well as your other systems. That's an important piece of this, right? We recognize, even myself as a Databricks employee, we want you on Databricks as much as possible. We but we recognize you have other systems in your ecosystem. So, that next layer that
[06:18] we're going to talk about is all open source and can be leveraged with other with other platforms that you're using. But when when we're talking about Databricks, we have to have tight integration with UC. It governs everything. So, we're going to um we're going to integrate with UC pulling
[06:33] in schema, pulling in metadata about your existing assets so that you can you can craft contracts intelligently so that you can govern your assets that already live in Unity Catalog. We're also going to use it as a system of record for for our quality, right?
[06:49] We're going to write out quality records there as well. That next layer is is the one that I I think is is is the core of our talk. You know, this is part field report, part you know, introducing some technologies that you can use as you try to figure
[07:05] out how to operationalize contracts or operationalize governance in your enterprise. And these, as I mentioned, are all open source. So, on the far left you have the definition layer. This is the specifications that Michael just mentioned, ODCS and ODPS. These were developed by the Bitoff
[07:21] Foundation and and and organization and may you know, promoted into the Linux Foundation open source. And they're just agreed upon specs for the industry, well adopted by enterprises to to define a common standard for how we
[07:37] express data contracts. The middle portion is a open source application that's part of our Databricks Labs set of projects and it's called Antos. So we're that's going to be a fair part of our demo today is showing the Antos application. I imagine there's probably a number of
[07:53] you that have haven't heard of it yet and so excited to get you introduced to that and it is our management layer where we create, govern, life cycle our data contracts and also can promote discovery. And then on the right side of that kind of middle of the sandwich is is our DQX.
[08:11] I'm hoping that a number of you have heard of DQX or hopefully using DQX today. Again, another one of our Databricks Labs project focused on quality and so we'll show how we've integrated DQX into the data contract life cycle that we're managing in Antos.
[08:26] And then the upper layers are around adding intelligence, right? Adding observability for for quality monitoring so that you can keep track of how your your your data assets have either, you know, are they complete, are they fresh, are they drifting over time? And so we'll talk about that a little
[08:42] bit and then no talk in 2026 would be complete without a little bit of GenAI conversation. So we'll we'll include the intelligence layer as well integrating with the Databricks model serving endpoints for foundation models. And we'll leverage that GenAI to help us
[08:57] with contract authoring and discovery. And then finally again, can't do a Databricks talk without Genie or AI BI. So we there won't be a huge feature of our of our demo or our presentation today, but it's worth noting that it's it's core to you know, to making use and and deriving
[09:15] value from these data assets that you're taking such care to curate. And it's worth calling out that while um Antos surfaces an MCP layer that you can actually access, you know, you you would have heard yesterday about Genie ontology. You can surface this Antos MCP
[09:32] to Genie as well to be able to connect the two. All right, awesome. This slide is really here just to wake you up, to be honest. Uh we're going to transition into the the demo portion. Uh and so I am going to pull up my other screen here. All right.
[09:48] Excellent. So this is Antos. Um again, open source application. Uh we'll be sharing the GitHub link for this after the presentation, so you can access it from there. Again, hosted as part of our Databricks Labs project. It's also available on Marketplace. So you can also access Antos and install it in your
[10:04] environment through the Databricks Marketplace as well. Um here you're going to see a lot of things. I really um oh the the other thing I was supposed to is zoom in on. We're going to start with the authoring process, right? So in this in this life cycle that that Michael talked us through, uh I'm not going to demo
[10:20] standardization. We'll talk a little bit about the spec, um but it wouldn't be a super fun to to just spend a lot of time looking at at YAML files. So we're going to we're going to fast forward to the authoring step. Uh and it might sound like a silly thing, um but when I first started on the Boeing account uh over a year ago, we we started working with
[10:36] contracts. We started implementing contracts. And you know, it's they're machine readable, that's great. Um but if you want to incorporate other personas into the process, um you you really need to to create an interface to be able to facilitate collaboration. So we're going to spend a little bit of time showing you that that
[10:52] authorship piece. So All right, coming back to to Antos. Michael will give us a little bit of a a demo of the the the larger discovery piece a little bit uh later on. Um so I won't focus too much here. We're going to go ahead and drill directly into our data contracts view of that of things.
[11:09] Um and you're going to see um you know, I've got some contracts already here, but let's assume that we have a new data set that we're onboarding, and we want to create a contract from scratch. I'm going to go ahead and remove this one that is a legacy from uh a previous demo. Uh and I want to go ahead and kick off
[11:26] um my my gene GenAI portion of this cuz you know, we're not going to sit here and let us you know, make you watch it spin, so we're going to go ahead and ask them to start creating a contract for us, and then we'll kind of in parallel do the same process manually so that you can see the UI, and we'll compare the
[11:41] results a little bit. So, I'm going to tell ask Ontos over here to draft a contract. I've got a catalog here conveniently named Safe Skies, which is the title of our talk. I've got a bunch of flight operations type data. This is data we we synthesized from just
[11:59] flight operations domain expertise. We've got, you know, fuel, we've got maintenance, passenger information, and then general flight ops stuff. And so, I'm going to go grab this 80 ADS-B raw table, and that's the one I'm going to ask for a contract for.
[12:16] Uh draft a contract. I know that's probably hard to see, but draft a contract for this table. And we'll let that spin a little bit just to, you know, kind of call out. It's going into UC, it's pulling the schema, it's inspecting the columns, it's sampling the data, it's doing some
[12:32] fun stuff to get an idea of what's in that table, and it will start drafting that contract for us. But in the meantime, I want to show off the UI a little bit as well. Um and so, you know, I can open up create a new contract here. I'm going to just call it what? Demo.
[12:48] Uh let's call it Daas Demo, right? Um not going to dive too deep just because for the sake of time on all the different governance pieces, um but you can assign these contracts to owning teams, projects. You can kind of scope it down to who has access, who has
[13:03] stewardship of this contract. Um so, I'm just going to pick some some values here for now. Uh importantly, you can assign this to a domain. So, we're going to dive deeper into this later around this idea of productization, um and so, this facilitates that, right? How do we
[13:18] create data products? How do we assign them to domains? How do we create a hierarchy around domains? Um so, I'm just going to assign this one to, I don't know, industry. Uh and then there's a bunch of additional metadata um that we can uh that we can add as well, but uh not going to
[13:34] spend too much time on that right now. But that gets us a contract. Um we have some metadata at the top. We have a life cycle that uh associated with this. Um where is it in it's in this case it's in draft, right? Um right now we don't have any schema. Um so I'm going to go ahead and you leverage that UC integration
[13:51] that I spoke of to um to pull in the UC metadata. So here's my Safe Skies catalog that I pointed out. I'm going to grab Let's just do the same table. Uh ADSB V2 raw and I'm just going to infer the schema from that table. And so it's
[14:07] going to go again, it's going to pull in that metadata and it's going to create the schema for us. We don't have to um we don't have to draft this from scratch, right? We don't have to you know, there may be cases where you're designing a data product before you have the asset created and that's fine, but in this case, let's not recreate the wheel. Let's just pull it in. Um so that
[14:24] gives us our schema. Within the schema, we can do things like add constraints, uh add quality rules, um add, you know, governance, uh classifications and tags, things like that. Um these are the types of things you may want to include in your your
[14:39] data contract. All of these uh fields, all of these um items that we're we're filling out in terms of information, again, we didn't dream these up. These are defined within that ODCS spec, right? So uh again, we're leveraging on um work that's
[14:56] already been done in industry to define what are the important things to capture as part of a uh that interface between your data producers and your data consumers. Uh and so we won't um we don't we don't need to, you know, belabor all the different aspects of this, but we've got uh we've got, you know, things like quality rules. We'll
[15:12] show off the linked data products in just a little bit, but quality rules we can add at the contract level as well. You'll notice this is kind of a preview for the the aspect of our of my demo, Um, but we can add DQX checks, custom SQL expressions to to define quality checks within the
[15:28] contract. This gives us a centralized place to uh to define what quality means for this asset and then make it discoverable for those who want to consume it. We can add team members. Who's responsible for this data product? If I'm, you know, looking down the road a
[15:45] little bit, if I'm a consumer, I want to know who's maintaining this. Who's Who's got ownership of this from both a technical and a business perspective so that I know who to contact if things go awry, right? So, we can log our team members, who's who's maintaining this data product. We can add add SLAs uh as well. Um,
[16:02] you know, how how What is our commitment from an uptime perspective? From a query response time? From a data freshness perspective? What are we committing to? Again, so that that interface is clear. Our our consumers know what to expect. As a producer, I know what I'm being held to.
[16:20] Uh, and the rest is largely metadata. We'll come back to some of these other aspects. Uh, I do want to just kind of go back to the lifecycle piece a little bit and just show off um the the you know, we can go in and request a data steward to review. So, this is again, the idea of curating data
[16:35] product shouldn't be a solo endeavor, right? There should be multiple personas involved. Uh, you should have your your data engineering teams, your data data teams helping to define the contract, but then you should have uh product owners and uh data stewards reviewing and and collaborating as well.
[16:51] So, I can send this request. So, they're basically saying, "Hey, I've got a draft. I want to I want to, you know, propose this as a as a new data contract." Uh, I'm just going to change, well, yeah, I can change personas. I'm hesitant to do that cuz I don't want to lose uh this over here right right now.
[17:07] So, I want to be able to show that off. Uh, but I can um start review. So, let's say I log in and I I've got the persona of a a data steward. I can review this contract. Uh, it it moves it down the life cycle status to uh under review and then I can add comments to it as well, right? So, I can
[17:24] add comments. Hey, we need to change this. We need to update this and we you this is a collaborative way of managing a contract instead of just uploading YAML to a volume somewhere, right?
[17:39] Um okay, and then I want to go ahead and change this. Uh and this is a good and that's going to be probably there. There we go. This is kind of a good depiction of of that contract life cycle in full, right? It's a draft proposed. Right now it's under review. Once the data steward approves it, then we can activate it and certify
[17:55] it. So, this gives us a way of um you know, you're going to have data products that are in in dreamland or in dev or in um you know, in the process of of being certified. Uh and then you're going to have those that you actually want to put a badge on and put a stamp of of
[18:11] certification around and this allows you to do that as well. Um I'm probably running long, so I'm going to go ahead. You can also um there's within ODCS, if you're not familiar, there's versioning semantic versioning around your contract, so you can also manage versions. So, that as
[18:27] you do new releases of your data product, that's captured as well. All right. I'm going to fast forward on that. I just want to kind of show off uh our ask on those did do what we asked, right? So, it went through the whole process of grabbing all the metadata, computing some column profile statistics, and then
[18:44] it drafted us a contract Uh in this case it's showing being shown um you know, it streamed the token to us in in real time with uh in a JSON format, um but then we can go and view this this contract in the uh in the the the UI itself.
[19:00] And so, you can see it generated a nice description for us, how it should be used, limitations. It did all the schema work for us. And conveniently, it also defined a bunch of quality rules for us. So, this can serve as the um the seed for our enforcement step.
[19:20] And then if I want to see what this looks like in practice, I can view the ODCS contract itself. This is kind of that native form of the contract stored in YAML defined by the Bitwave Foundation organization and uh can be again machine readable can be
[19:36] shared across across systems. Okay. See if I'm missing anything on that. Think we're good. All right. So, I'm going to just really quickly switch back to the slide where All right. So, we have a contract now. We've gone through the pains of defining what
[19:52] this data asset should be, what this product should represent. Now, that's fairly meaningless if we don't do anything with it. If we don't actually main you know, maintain it and and actively enforce it. And so, we're going to talk next about how do you actually integrate this with your data pipelines
[20:08] themselves, right? Um it can't sit the the contract itself can't sit in a silo. We need to integrate it directly into where our data is being moved.
[20:25] All right, perfect. Um All right. So, the enforcement layer is going to be driven by DQX. I called this out a little bit earlier. DQX is the Databricks Labs quality expression engine. If you've ever used anything like Soda or Great Expectations or anything like that, uh it's it's pretty
[20:40] similar and it's built to work directly with Spark and with Unity Catalog and on Databricks. And so, this is going to be kind of our enforcement layer. I included this just because again DQX has been around for a while. It is very mature. It's very well adopted at an enterprise scale. But,
[20:57] here are some new features that you may not be aware of. Importantly for us for the sake of this conversation, in December of last year, Um, include it it they added direct ODCS connectivity or support for ODCS to grab and parse rules from a contract. And so, uh, that is
[21:14] worth calling out and then, uh, they've added a bunch of AI support. AI assisted rule generation, um, uh, let's see, uh, email anomaly detection for for records. Um, so some really cool features in here, um, worth worth exploring as you, uh, look
[21:32] look into implementing quality at scale. All right. Did we switch over? Perfect. Okay. So, I'm going to come back. I'm going to choose a contract that I've already published. So, let's just go with, um,
[21:48] let's see. Let's go with this flight schedule. And I just created this button for the sake of for the sake of demo. Really, what the way that I would expect this to be used in practice and the way that we use it is you're going to, um, you're
[22:04] going to run a workflow as part of your data pipeline and you want to be able to integrate it back into um, where your contracts live. And so, of course, um, as you would expect, Ontos is got a fast a fast API back end. It's exposing the, um, the endpoints as part of uh, as part
[22:20] of its back end service. And so, we can call into those as part of our our data pipelines. And so, I'll kind of show that off. I'll click this button. It's basically, you know, kicking off a pipeline for me. Um, and so, you can imagine in your, you know, data product bronze to silver, silver to gold
[22:36] pipeline, how you could, um, you could do the same thing. This takes a little bit to run. Um, and so, I'm going to go back and I'm going to just show you a a job that I ran earlier. And there's some of this that I'm going to skip past pretty quickly. This is
[22:51] this is the only part of the the demo where we're going to show a little bit of code. So, hope that the, you know, the being a technical audience, I I hope this is okay. Um, the important piece here, number one, we're going to authenticate with Ontos. So, it's going to authenticate with the run as credential
[23:07] that's that's running the pipeline. In this case, it was kicked off as the app, so it's using that credential. But, if I was, you know, with a part of a different data product team, I can authenticate as that product team into Ontos, so I can only pull contracts that I have access to. Importantly, here's where we can call
[23:23] that API endpoint that I just showed on the Swagger page to be able to pull that contract itself. So, this is going in and you got the centralized repository for contracts. Now, by contract ID or by table name or by whatever
[23:38] search criteria you want to use, you can go pull in the relevant contracts for this data pipeline, and it's going to load all those quality expressions. Here, we we use the DQX DQ generator class to to be able to convert those
[23:54] ODCS rules into rules that DQX DQX knows how to execute against the data set. And so, this is giving us our our rule set that is going to be executed as part of our pipeline.
[24:10] And now, for this for this demo, you know, some of my contracts have one table, some of my contracts have multiple. So, for the sake of this demo, I just iterated through every every what's called schema or it's it's really a table in the in the contract to
[24:26] be able to execute the the rules against each table. So, that's why you're going to see this for loop in here, but basically what this is doing is it's using that DQX. This is another DQX class called the DQ engine. We're going to initialize that, and then this is what we're going to be
[24:41] used to execute the those those preloaded rules. And so, this is essentially what it looks like. We just do that use that engine. We've we've read in our table. I'm going to zoom in here a little bit. I know that's probably pretty small. So, we've we've loaded in our table. We're We're use this engine to apply
[24:56] checks by metadata and we're using those rules that we grabbed from our contract to execute it. And you know, I can I can do a little bit of analysis here. I can grab um I can grab counts. I can look at how many errors do I have. This is, you know, resulting in a data frame that has additional columns for errors and
[25:12] warnings and, you know, calculate kind of aggregate metrics. All right, I'm going to try to zoom through this cuz I'm probably behind on my time. Um This is just showing us what rules it grabbed. Essentially, right? So, we've got, you know, hey, I need to make sure
[25:27] this Ical code is is is valid, right? Is is not null in this case. This one is actually using a SQL expression to make sure it's in the right format. Uh so on and so forth. We've got a handful of rules here. Those are all being executed against. It shows us we have 24 that errored out. Pretty
[25:44] much every record that I have in this table also had a warning, so you can have severity levels. The reason I have this warning is cuz this is intended from a demo perspective to simulate flight sensor type data and this is showing me I I have a freshness issue. It's expecting records
[26:01] updated every you know, within a certain amount of time and because I'm not, you know, continuously simulating this, I've got this warning. But here we can see a sample of like the fail the failure records. Um And so, what was our failures? It was around the barometer metric the barometer
[26:18] reading not being appropriate for the data. And so, we can see an example of those those error records. Uh and we can get a summary metric of like how many records were quarantined versus how many were passed through in a valid state. So, the way DQ actually would normally use DQ X is you would
[26:35] pass through valid records to your clean valid silver data set and your your invalid records you would quarantine to figure out what to do with them later and have a process for handling those. And so, that's kind of what you're you're seeing here. Um importantly or interestingly,
[26:51] uh this this metric is also getting pushed. Uh post metric is is the function I created, and this is actually calling back out to that Ontos API. So, we can come back in here, we can go to our data contract, and we can actually scroll down here, and we can refresh this data
[27:06] quality tab here, and we can actually see that that run uh published information back to Ontos. So, that um you know, this is not necessarily intended to be the end-all be-all for your quality reporting. Um you probably have additional dashboards for that, but it's helpful information for your data consumers, right? To know,
[27:22] "Hey, there are quality rules being executed. What what's happening on this data product?" And so, this helps to to drive that enforcement layer. With that, I'm going to turn it back over to Michael to talk a little bit
[27:38] about the uh the discovery part. Thank you, Alan. Yeah. So, as we were been talking, uh we've been focused a lot on contracts, and uh kind of avoided or didn't really deep deep dig too much into the data product. Uh just like contracts, there is a spec a a
[27:55] companion spec for products, and it and it does a similar thing. It creates a consistent definition across the enterprise, across your organization, of what a product should be. It's actually sometimes quite difficult to get everyone to agree. So, it's very good that we have this.
[28:11] Uh and the way that it's uh probably first explored and experienced is like a consumer coming to Ontos who wants to find data. And so, that's something we do Oh, I'm sorry about that. Thank you. And uh that's uh what we Ontos has a feature here called the marketplace, and unlike
[28:27] the views we're seeing before, this is a a roll up kind of a higher level view where only curated, active, and published uh data products are showing up. So, this is a good landing spot for those consumers who are looking to explore find data. It has all the the usual bells and whistles. I think you
[28:43] guys have probably seen contracts uh sorry, uh catalogs before. Uh you can search. Of course, there's Ask Ontos over here. You can use AI to help you find data that you're looking for across your your enterprise. And there's also uh ways just to filter by domains, so you can uh
[28:59] dig down into just a specific uh domain that you're interested in. But, let's take a look at this uh product here. And just a few things I want to highlight. Um so, one one thing just to take a one
[29:14] more step back and just explain, with products uh the product spec, the main goal is actually as a wrapper around the contracts. So, you can think of the product as kind of the container. It has a lot of business metadata and other contexts. Uh
[29:29] and I'll try to highlight a few very important things, but it's also holding all those contracts, kind of like the cargo inside the shipping container. Um Okay, we will not do that. Uh And and that's why when I scroll down here for this data product, I'm actually seeing these deliver rules, which was
[29:45] actually the the contracts with the data itself. Uh but, the first thing I really want to talk about and the main thing to cover here is how does this help um in in um create trust across the organization? And that's where, just like with the uh data contract actually, we have a few
[30:01] things on the right here. The first one, and probably the most important, is the owner. We all probably know that, but you know, without a data product owner, no one's going to touch that data, right? You need to know who's responsible. Someone has taken taken on that title. Uh also, just also like the contracts, there's a
[30:18] status. And we're only looking at active ones here, but there is a full life cycle status built in uh or flow built into both Ontos and the spec uh that you can look at here and be like, "Okay, this is a an active development. It's something I can trust." And another layer that is
[30:34] uh added added layer of confidence that we can add is we can certify. And so, uh this is maybe something that's done uh in a different role in your organization that's going around saying this is a a a true source or the and you can have different levels of certification. I'll just do this to show it, but that's one
[30:50] more way you can give confidence to the consumers that this is the right data source to be using. Uh we don't really have unfortunately time to demo this part, but it's something that we really use and excited about in Boeing and it's the other most requested feature is probably how do I get access to the data.
[31:06] And so there's a great request access workflow and UI built in here that we then can automate and use for the actual access management. Uh this also gives us traceability and auditing on how that works and you know, as a big enterprise we really enjoy all that as well. But I'm going to sidetrack that for a
[31:22] bit and I want to highlight one more feature that's very important and we're going to talk about how this creates a trust loop for the consumers from and the producers. And that's this subscribe workflow. Now you can customize this in many different ways and in our case we
[31:38] actually have a full terms of service that we're going to accept because uh by subscribing I'm now wanting to consume this data and I have to be careful that I'm using it in the right way. So let me check through all those. I have read everything very carefully and then this is going to go to the team
[31:55] but I'll give just cuz I want access. I'm sure you get lots of great requests in your organization. All right and we're done. So that's going to be very important partly to the producers. Now we know
[32:10] another view of who's using our data. We do have lineage and we have other ways but this is just one more thing. You're going to see interest and people who are stakeholders of the data and then it helps you understand that blast radius of who you're affecting when you make changes. And you can also
[32:25] reach out to them and engage with them. Okay. I think did I miss anything there Alan? I think that's all I want to cover in the discovery. We're going to now jump to the maintain section which is where we're really excited about how we can
[32:40] really take everything we've done here, all this governance work, but actually turn it into code, into automation, and something that we don't have to baby and hold hands hold the hold its hand. This is where we're going to see a really cool few cool features in Antos
[32:57] um that uh that help us as an organization uh maintain this growing mesh of data. So, a lot of times we're trying to answer when governing things, we're trying to answer does the contract still hold? When something breaks, how are we going
[33:13] to tell everyone who how how are people going to find out? And is everything governed as we define it, as we want to define it? And so, one of the things that can happen, and this is probably the worst-case scenario as a consumer, what's the worst thing that can happen? The contract breaks. Someone did not
[33:28] follow the process, they didn't branch off to a new version of a contract and communicate that we're deprecating the old one and migrate people over, or maybe there's a mistake in the code, but something changed in the data itself and now we're not conforming to the contract anymore.
[33:43] And so, how do we how do we deal with that? Well, at least we can check and catch it. And so, I'm going to show how Antos makes that really easy. So, so I have to jump into the contract to do this, but a few Oh, I keep forgetting to do this. I'm very sorry. Thank you. You You didn't see You didn't miss anything.
[33:59] Oh, I didn't switch that time. Oh, I went to the wrong one. Okay. Uh so, this is the part that Alan kind of glossed over when he was looking at the contract, but very very near the bottom in Antos, we have this feature called uh source confirmation. And what this is doing, and because it's
[34:14] integrated with Unity Catalog, it's going to go to the source data, in this case Unity, and then compare it against what the definition and the spec is saying the contract should be. So, I'm going to I'm going to This is something we we automate and we want to have running in the background. This is how we do that governance uh as code,
[34:30] but for the sake of the demo, let me just trigger it. And oh my gosh, there's been a there's been drift. The schema no longer matches. But if you remember just a few seconds ago, I subscribed. I'm also a subscriber to this data. And so not only will this notification go to
[34:45] the producer, all the subscribers or all the consumers will now also see this that there's been a breaking change to the data, and they can react on it. So that we really think helps add that layer of trust and accountability and a and a and a loop feedback loop on um
[35:01] on the data. And another really exciting feature in Ontos, and one that we're just starting to use and there's a lot of potential here, is this whole compliance uh engine that's in here. Uh and it actually allows you to write compliance or queries uh as rules as
[35:17] like policy as code. Uh I won't go into that all the details of creating one, but there is a a language um that's uh a query language as part of Ontos that you can use. And in this case, what we've done is said, and this is because we're uh still, you know,
[35:33] rolling out contracts, but we wanted to know how many uh data assets in the organization actually have a contract. And of course, we want this to be 100% and and never fail, but in our case, uh we are we are we are not all the way there yet. But what this does is that we
[35:49] can run this periodically, we can check. It's a metric we can follow now. We can make sure we're actually lowering this number and that we are following up with all these data producers who are down here in the failed section to see what's going on and help them get on to Ontos and get a contract backed
[36:05] out. So there's just a lot of potential here, very, very flexible because you can pretty much write anything you want in here, uh and it'll it'll query all the Ontos uh details for that. Um
[36:23] Great. Okay, so very quickly, there's just one uh one gap that we have to admit here that we're working on closing. Where Ontos and all the stack that we're showing you just doesn't quite get there. And this is with statistical drift. And but there is a really great product and so we're just going to show
[36:39] it so that everyone's aware of it. But we are using Databricks data quality monitoring for this. And so this is already built into Databricks, natively works, and it's just been a really really great confidence boost for teams consuming and to catch those really tricky data
[36:56] data drift issues. What we are hoping though is that we will be able to extend this both into Ontos and maybe into the spec itself so that we're actually all doing this in one place and then can even start writing compliance and governance off of that as well.
[37:14] Okay. So that's it for our demos. Now, what do we do next? Let's do our call to action here. We're just beginning as well, but first for you guys who maybe haven't started with contracts at all, haven't explored Ontos, start small. One domain, one
[37:29] contract, one published product, one quality rule. And then remember, a data contract is it isn't a document you sign. It really has to be a system you operate. It has to grow with you. And then just an extra highlight, we got to standardize.
[37:45] Move away from your wikis and tribal knowledge and PDFs that are floating around to a version controlled agreement using the open standard. And then as we've been showing, automate automate everything, especially the enforcement. Quality, like Alan was showing, really belongs in the pipeline
[38:02] and it should fail before it gets to your consumers in your gold layer. And then use Ontos or a tool like that to create policies and tooling to maintain compliance over time. And then make it valuable. If people can't find the data,
[38:17] it's not going to be very effective. So make it discoverable, use the contracts and the data products back us to form a marketplace and help bridge the gap between data consumers and producers and discovering data in your organization.
[38:33] That's me. Yep. So just calling out one more time that this is the extent of our of our our sales pitch if you want to call it that. We're not selling anything except that these we just want to highlight some tools that that are available for you to use if you
[38:49] want to build something similar. If you want to integrate these pieces into your stack. Everything we showed today was was open source ODCS ODPS as we mentioned the specification itself. We've included the QR codes here for you to to go and and access those repos. DQX the the quality
[39:06] engine and then of course Antos on the end. I will be I guess make a confession and be transparent here. I've added two repo links for the Antos piece. There is the the publicly available Antos. This the the team that is developing Antos which is
[39:22] not currently us. We are working to contribute back and and add things feature sets to Antos but it's it's not our brainchild but the team that's developing that is very very active. It is evolving quickly. They are throwing features into Antos and it's amazing. But we also did a fork of it
[39:37] for for our presentation to add some of the features that we cared about and so if we'll be working to get those as I mentioned back into the open source repo. We'll be submitting those feature those PR's back into the main fork but if you're interested in seeing some
[39:52] of the specific work that we've done I added that link as well and these of course will be these slides will be uploaded. Um Yeah. Yeah. So I guess one more sales pitch. You know, the journey has just begun but join us. Adopt the open standards. Use
[40:08] DQX and join the Antos community to shape and govern what's next in this space.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.