Skip to main content

Data Contracts and Products: The Foundation for AI-Ready Data Platforms

Summary

  • Organizations using data contracts are 3.5 times more likely to successfully put AI into production according to BARC survey data presented in this video, yet most AI failures trace back to poor data quality, ungoverned ownership, and missing schema documentation.
  • Open standards including the Open Data Contract Standard (ODCS) and the Open Data Product Standard (ODPS) provide a vendor-neutral contract layer with three pillars: linking producers to consumers, connecting business context to data, and surfacing meta-metadata above any lakehouse architecture.
  • Lufthansa Cargo applied these principles in production by unifying over 100 real-time data sources into a governed foundation using the Actian Databricks platform, demonstrating how open standards prevent vendor lock-in while enabling interoperability.

Data Contracts and Products: The Foundation for AI-Ready Data Platforms

Watch: Data Contracts and Products: The Foundation for AI-Ready Data Platforms
Organizations fail at AI deployment not because of model quality, but because data is uncontracted, unowned, and ungoverned. Gartner expects 40% of agentic AI programs to be canceled by 2027. Yet companies using data contracts are 3.5x more likely to put AI into production successfully.
Learn from Jean-Georges Perrin how open standards like ODCS (Open Data Contract Standard) and ODPS (Open Data Product Standard) provide the missing contract layer above any lakehouse architecture. Discover the three pillars of data contracts: linking producer to consumer, business to data, and surfacing meta-metadata. Explore how Lufthansa Cargo unified 100+ real-time data sources into a governed foundation using the Actian Databricks Data + AI Platform, and how open standards prevent vendor lock-in while enabling interoperability across your data stack.
🤝

Chapters

FAQs

What is a data contract and why does it matter for AI deployment?

A data contract is a formal agreement between data producers and consumers that defines data quality expectations, ownership, and schema governance. According to BARC survey data cited in this video, organizations using data contracts are 3.5 times more likely to successfully put AI into production because they address the root causes of AI failure: poor data quality, lack of governance, and unowned data pipelines.

What is the Open Data Contract Standard (ODCS)?

ODCS is an open standard, now hosted by the Linux Foundation, that originated at PayPal and provides a YAML-based specification for defining data contracts. It covers three pillars: linking producers to consumers, connecting business context to data, and surfacing meta-metadata that makes data trustworthy and machine-consumable for AI workloads.

How does Lufthansa Cargo use data contracts in production?

Lufthansa Cargo unified over 100 real-time data sources into a governed data foundation using the Actian Databricks platform, applying data contract principles to enforce ownership, quality, and schema governance. Using open standards prevents vendor lock-in and enables interoperability as their data stack evolves, which was a core requirement for a global logistics operation.

What is the difference between a data contract and a data product?

A data contract governs the agreement between a data producer and a consumer, specifying quality, schema, and ownership obligations for a dataset. A data product, defined by the Open Data Product Standard (ODPS), packages data as a reusable, self-describing asset discoverable and consumable by both humans and AI systems; adoption of data products has reached 69% according to figures cited in this video.

Full transcript

[00:08] So, thanks for being here. This is compulsory things you've got to read before you get in. I'm just kidding. Uh but you've got to take the survey. Okay? And if it's on five stars, give me five stars. If it's on 10 stars, give me 10. You know the drill. Okay. Uh let's get
[00:25] started. We're about to talk about scaling AI and why data products and data contracts are going to help you in this journey. So, my name is Jean-Georges Perrin. And uh
[00:42] yeah, it's completely unpronounceable unless you took French in high school. So, I mostly go by JGP since I moved to the US. I write some books. Um my first book was actually my first published book was actually Spark in Action at Manning. And
[00:58] I I thought that because we are here in the art of Spark, it would be a good idea to bring one of the few copies I had left and and give it away. So, Brandon knows the drill. The first smart question after the after
[01:14] the talk gets a free book. And if you're not into Spark, uh I I brought what my latest published book, Implementing Data Mesh. Okay? So, um I'm not here to to sell my books or whatever, but I just to say that if you've got a boss and you've got to
[01:29] explain things, the books for kids are really super popular. Okay. And they go and the profits uh the proceeds are mostly going to a charity. So, uh so that that's about me. So, let's get in let's get into uh in in in into um
[01:47] into the subject into the topic. So, scaling AI, putting AI in production is a risky business. All right? And it's not me saying it. This is figures that are coming from MIT, McKinsey, and Gartner. And they say that, well,
[02:04] it's it's it's a tough it's a tough game whether you want to have GenAI or deploy AI agents, etc. Uh so, what what I've been working I I work directly with our CTO, and we we've
[02:20] we've isolated really four causes four reasons for this failure and this lack of maturity. It's and it comes to really the the fact that we don't It's not, you know, everybody's talking about context,
[02:36] and true, context is something that is really important, and we'll we'll talk more about context, but even poor data quality, okay? I remember back uh back when GenAI came out when back when ChatGPT came out, people were saying, "We've got AI. We don't need data."
[02:54] And then they just realized, "Oh, we need data. But, oh, we don't need high-quality data because AI is going to correct it." Well, we know where we are now, right? So, it's it's all those things data quality, data governance, it's really impacting how we are actually how we actually
[03:11] doing data. And this is why I I strongly believe that uh data contracts and data products are here to help. Data contract is a promise to to give you the data according to
[03:27] data quality rules, according to SLAs, according to description to high level of description, and data product is a delivery mechanism for this data. And they are really intrinsically linked.
[03:46] So, we did we did a survey with BARC. BARC is a prominent um analyst in Germany, and they are see see them as a Gartner of Germany and they're really really strong on data on the modern data aspects of things. So, they've they've run a survey
[04:03] uh and I'm going to share some of the results with you. So, and and what we discovered is that and we were really surprised, okay? We we didn't know that. We didn't anticipate this thing is that people
[04:20] using data contracts and data product are actually almost three and a half times more successful uh put putting things into production, at putting AI into production. All right. It's not 3%.
[04:36] It's three and a half times more, okay? So, people having like, you know, three projects in production are the one with data products have 10 10 in production. And it's Is it Is it a correlation? Is it a causation?
[04:51] Because we were really surprised by the result. We we don't know that. You've got to wait for until next year if we run another survey. But, it's really it's been really impressive in terms of numbers. And so, how do we how do we get started, okay?
[05:08] And this is where I'm going to try to to help you get started. So, if you if you look on on the on my little uh uh desk-ish lectern over there, there's a small device there that is counting the
[05:25] numbers of users on an experimental platform. So, if you want to join this experimental platform, feel free to to scan the QR code, fill a little of few things, and the thing here should actually do something as you're doing it, okay? So, so it's about
[05:42] creating a community of practitioners. Okay. So, who doesn't know what a data pipeline is? All right. All right, I'm a good crowd here, so I don't have to re-explain it. Um but it's really, you know, when we're
[05:58] doing data data pipelines, it's really about bringing the data from the producer to the consumer and in the process, there's quality assurance, data quality, and there's documentation. In my career, I've been really lucky to
[06:15] have seen a lot of lot of these project, lot of these pipeline with the correct documentation. And it was delivered with the correct documentation, and the documentation was valid for like maybe 3 4 days after delivery. Okay? And and when AI comes into place as a
[06:33] consumer, well, the pipeline is a lot more complicated than that. And Oh, someone signed up, you see? See what happens? Okay? Um So so uh when So so when you feed AI, you've got to give a lot more information about
[06:48] that. Okay? So let's let's drill a little bit into that. But before that, I wanted to share you a little story, a little personal story. I have four kids. One of one of them just called right now. But the thing is So this is uh
[07:04] this is a story that is with my older one. And he works in in a major bank and he's overseeing this this data pipeline, which is feeding an internal agency that he's he's doing uh regulatory reports. And
[07:21] as you can imagine, most of the time it's okay. And once it was not okay anymore. And you know, some of you are up roughly my age and have young adults as kids. You when they call you,
[07:37] it's usually not a good sign, right? They need money, they have a problem, something, okay? So, he called me. And he said, "Dad, this not this is not working anymore. There was no data. I'm going to lose my job."
[07:53] And I told him to investigate and what he found out was actually that the upstream system was down. I'm sure it never happened to you, huh? So, I told him, "Look, if you had data contracts at this two stage, you would
[08:09] actually know before your consumer that something wrong is there, okay?" So, so that that's really that's really that's really the story. Uh and of course, you can imagine as a dad, I was a little bit sarcastic in my
[08:24] delivery message. Did he implement it? Only only the legend knows. Okay, so So, this is really about I have a data contract you you cannot refuse, okay?
[08:39] And uh so, what is really a data contract? So, it creates for me it's this this the three pillars. It creates a link between a producer and one or many consumers.
[08:55] It creates a link between the business, you know, the business with a capital B we we always deal with in the company, and the logical representation of that data, and the physical representation and the implementation of the data. So, that's a second link. And it describes
[09:13] the meta metadata. So, what I define by so so funny anecdote there. Uh the first time I I went I did this deck and I sent it to my marketing department, they removed one meta. Poof. Uh
[09:29] but it's really about meta metadata. Everybody here knows what metadata is, right? Column names, table names, column type, etc. But what's the meta-metadata? It's really about uh It's really about the the behavior of the of your data. The
[09:44] data quality rules that is that are associated to it. Okay? The SLAs. I work in a major financial institution. If you look at my LinkedIn, you will see which one. But the thing is, when I joined, I asked them what the
[09:59] SLA was and no one was actually capable of telling me what the SLA on the data was. Okay? Ask Ask your colleague when you go back. Okay? You'll be surprised by the answer. So, based on these three pillars,
[10:14] the the the natural conclusion is that the data contract is a source of truth for your metadata. If anyone disagrees with me, there must be a ring somewhere.
[10:31] But it's really When you think about that, it means that it's not in your Confluence page that was updated 6 years ago. It's not in your It's It's not in your database schema that someone can actually change
[10:47] overnight. Okay? It's really in your data contract. That's your promise. That's what you're giving your customers. So, And And you know, it's not One of the first time I was talking
[11:02] about data contract, someone was telling me that "Oh, it's something you're making up. Okay? There's no real problem you're solving." That's only six of the problems I'm actually solving with because I just wanted to be nice with people and not list all the problems you're solving with data contract.
[11:17] But one of my favorite is keeping normalizing and keeping documentation. Okay? Because you don't document anymore. It's in your contract. You're managing schema drift really easily because
[11:33] it's you know what when something happens. So, and you can also think as a product because you you're not doing data contract and it's frozen and you'd never touch it. It's a living document.
[11:50] So, and and in this survey, what we also find out is that 61% of this organization were actually doing data contracts, which for me is is an amazing figure. Uh but it really means that it's really
[12:06] it's it's really uh it's really taking off. it's really ugly on my screen here and I think it's really ugly here as well. Uh and I'm I'm more I'm more than happy to
[12:23] send you a a nicer version. I don't know what happened here. The When I was at PayPal, we implemented the first data mesh and one of the outcome of this data mesh, we realized that we needed the data contract.
[12:39] And we open sourced this data contract and this open source this template this open source template we created became the open data contract standard, ODCS. It's a little it's a Linux Foundation um project. It's It's kind of
[12:56] a little bit eaten at the bottom of the screen there. Uh it's governed by uh a bunch of uh consultant, software vendors, and end users. If you want to join below, the door is really is really open. And we actually
[13:12] took this this notion of data contract and we standardized it. And so, what I want to show you there is that there's a lot of things you can do with data contracts. You can And really the two things that are compulsory is
[13:28] the two things in red which is some fundamentals like the name and ID version and the schema of your data you're actually describing. All the rest is optional. All the rest as you fill it, you get more value from your data contract.
[13:43] Okay. Um And as you can see here um the third one which is completely blurry is uh is is the context and the context is what AI just loves, okay? Um but you've
[14:00] got data quality, you've got pricing, etc. You don't have to fill them as you're doing a data contract. You just have to if you want to do that if you want to do let's say if you want to define the support channel for your data well, this is how you can actually do
[14:15] it, okay? So, that's that's really the guidance we're providing the framework you can use and reuse to define and to build data contracts. And this is a standard that is massively growing. Um I did a start I did I did a survey
[14:31] uh as as part of as part of the being the chair of BDTO. Um back in March 2025, we only had nine companies doing it. As of May 2026, so really recent figures, we jumped to 114 companies.
[14:48] And there's a talk um from Boeing on Thursday morning about how they put it into production as well, okay? So, so and I don't have any shares in Boeing that I know of. One way to to see also how these
[15:04] contracts are actually impacting, we counted the numbers of employees in the companies using them. And as you can see in 2025 last year, we were roughly at 1.1 million. We are more than 4.5 million now.
[15:19] We um we we were present in three countries. We are now in 21 countries. And I must admit that it's mostly Europe and North America. Uh, APAC is coming is coming strong, but uh, still a bit lacking that behind.
[15:36] And one way to measure popularity of open source project is counting the numbers of stars they have on GitHub. And combined ODCs and ODPS have uh, more than doubled in less
[15:51] than a year. So, this is this this information I I I'm I'm not collecting this information for the sake of collecting it. It feeds catalog, okay? And and even if the quote from from Dr.
[16:09] Decker seems a little bit almost I would say almost naive, but it's a it's a data catalog that works. Okay? A lot of data catalogs I've been using and I don't know what's your experience with it. I've been filled by
[16:24] crawlers that actually bring no value. Okay? This is a catalog that works. So, how do how do we implement a a data contract, okay? So, we we picked YAML.
[16:40] You know, if you if you're if you're if you don't like YAML, you can do it JSON, you know, it's kind of interchangeable. Uh, because you know, machines understand semi-structured documents really well. Um, look at all the MD files we're
[16:56] feeding our LLM. And uh, and it's it's it's uh, it's something that some humans, I'm assuming that most of you in this room can actually write a few YAML lines. Uh, and it's a language agnostic. And the language agnostic thing I I'm always I
[17:12] always kind of was was joking about that and saying, "Oh, I'm sure someone can do it in Pearl." Uh and and someone actually did it in Pearl. So, flashback, okay? So, for for for some of us. Uh So,
[17:28] here here is what it looks like, okay? So, basic basic YAML file, you see a table name there, very nicely named. You see a column name, uh and you see a lot of textual information there. Okay? And when you see this is this is
[17:44] the richness we're bringing to to to the agent, to the consumer of of those uh data contracts. Another example, okay? Uh here you see that the logical type is a string,
[18:00] which is a business reference of a string, but it's varchar it's implementation is a varchar of two. If I'm a machine, I like the idea of a varchar two. If I'm measuring a drift, I need to know that there's a varchar of two. If it's a but if if I'm a data scientist, knowing that this is a string
[18:17] or making a report, a string is plenty enough, okay? So, there's this this logical type and physical type. And you see also this block there called authoritative definitions. Those are as I said, the data contract is a source of truth for your metadata,
[18:34] but it doesn't forbid it to actually outsource or reference more more information like here, a business definition that could be in your business glossary, or a reference implementation in GitHub.
[18:49] And many more types, actually. One thing I really like uh about uh about um about uh data contracts as well is the notion of being able to define your team in the data contract, because this is often something which is
[19:06] kind of a tribal knowledge, uh con keep kept secret in the in the in the company. So here you clearly see that this initial guy called C. Eastwood uh was replaced 2 months later by this guy called J. Wayne because he was
[19:21] playing it dirty. Okay, some people, usually the people over 40, get the joke. If you didn't get the joke, well, you can visit Alcatraz. Uh I think there's uh
[19:36] there's there's a movie there as well. And the thing is, you can't explain a joke because otherwise it's not a joke anymore. Okay. So, how do I get value? I'm obsessed about getting value quickly because you know, we are not in this era anymore
[19:51] where I can start a data warehouse project and tell my boss, "Hey, you'll get value out of it in 3 years, okay?" This is not the era we're in anymore. We need to be able to provide value very quickly. So here here is a way I I I
[20:08] provide value. Okay. Imagine that the little squares there are actually in a database because that's what should be on the screen. It's actually the schema that is guaranteed here by by my data contract.
[20:24] Now my schema is changing, okay? And I say, "Oh, okay. Um there's a drift. Is it is it I'm I can measure the impact of the of the drift. And in this case, it's a minor change, so I I actually modify my contract to match what my what what what
[20:42] has been done. And when I modify my contract, well, what I'm doing is I'm pushing it to um GitHub because this is where in source control your data contract live. And then when it's there,
[20:58] I can trigger automatically GitHub actions to actually populate my catalog, my observability tools, or any of these yes compliant because it's an open standard. Okay? And this is making sure that what is in production is what is
[21:15] being documented. Okay. So this is a question you've got to keep for yourself as you're as you're going as you're going to to implement data contract.
[21:35] Let's let's briefly talk a little bit about data products as well. So I'm pretty sure we are roughly like what 100 150 people in this room. I'm pretty sure that if I ask if we ask you to you know get a piece of paper out and write what the definition of a data data product is, we will probably get about
[21:52] double uh your answers to to identify that. So about a year ago, a little bit over a year ago, at a conference, we managed to put 40 people in a room uh and we discussed what a data product
[22:08] was. And this is this is a definition. This definition is a fruit of 40 people both from data engineering and product to actually define what was a definition. It's a long definition, but
[22:24] it made it to Wikipedia. Because that's a definition that now you will find on Wikipedia. If someone ask you, hey, what's a data product? You can point them because a lot of these industry people actually did that. But some of the highlight is that it's it has clear
[22:39] ownership. Okay? It contains a lot of metadata. It's uh it's either aligned to a specific domain or a use case. All right, doesn't have to be only a domain. And it adheres to fair
[22:55] principle, which is findable, accessible, uh interoperable, and reusable. For the architect geeks among us, this is a little bit like a what it looks like. Okay? Uh you've got data, which is
[23:11] interoperable data. You've got uh my famous pipeline here, you know, my data my data onboarding {{}slash} ETL pipeline that is a that is there. And you see input ports and output ports.
[23:26] You see these little triangles? Those are my data contracts. Okay? My data contracts are actually guaranteeing that what I get inside is what I'm expecting. Just imagine a factory plant in a in a in a in a car factory where you've got
[23:42] these spindles of steel coming in and you sample them to sec to make sure that they're the the right ones. Okay? And you've got contract at the output port, which is actually ensuring your delivery. And you've got standardized
[23:57] uh services that expose control, observability, and dictionary. So that's that's a little bit what's inside a a data control a data product. And so uh at
[24:17] same problems on the slide before, uh we define also the open data product standard, okay? And this is really what's what it looks like. This is a little bit more red here because what it needs it needs one out at least one output port with one data contract,
[24:33] okay? So that's why there's a little bit more red, but otherwise the fundamentals product information. If you want to define management ports as I just said, you there's a way to do it. If you want to define context for AI, there's a way to do it. Same thing for tags, uh
[24:50] custom properties, etc. Okay? So very very similar organization of the metadata. And And that's also, you know, same same bar study, the usage in in 13 months, okay? So,
[25:07] from late 2024 to early 2026, jumped from 48% to 69% of companies using um data products. So, it's not It's not anecdotical anymore, okay? It means that the majority of companies are actually doing a data product.
[25:29] So, also a question you've got to keep your to keep in mind. And And the whole idea is why are we doing all that? Well, because we want to have data ready for AI, okay? We don't care about humans anymore. We
[25:44] want to feed our AI with nice data. So, let's look Let's dive in a little bit into that. Before, when we were doing analytics with humans, you and me, um we required data. We had data contracts that actually
[26:02] defined the guarantee. And we had a source of of explanation where the metadata was, which was either a catalog, a glossary, or a marketplace. And there was a lot of, you know, all of the this knowledge we
[26:19] would bring in to use the data. The studies we did, uh the college we went to, the the the business rules in the company, the exceptions to the business rules in the company. All that is what we were bringing as human. And this data was
[26:34] also distributed by output ports. So, there's a lot of people saying, "Oh, AI-ready data is completely different." Well, I would tend to disagree.
[26:50] Performed by machine instead of humans, the source of explanation, the AI does not really need to go to your catalog, point and click trying to find things. It needs API. It needs normalized payload that is coming back and those are data contracts.
[27:06] And it needs the experience. And this is the context. This is the famous context we're all talking about. This is the context. All you see all this all the stuff is the degree we have, the work experience, all that as a context
[27:23] specifically for our favorite AI. And all that, you know, distributed by output ports, so we can actually combine them into a data product. Okay, so you see where data products are
[27:38] going and and the maturity that they're actually bringing to your data. So, are you ready? Are you ready for for that? So, let's let's look a little bit switch a little bit on trying to understand
[27:55] where we and Databricks have a value. So, we were talking about a catalog. Databricks has a catalog. And you know what common things between our catalogs are? We both spoke ODPS. Okay, so you open data product standard.
[28:11] You you need to be able to surface observability. We have a solution for that, which is Spark based. And all all that can actually be surfaced into the marketplace. Whatever your environment is
[28:28] and what whatever what whatever you're using underneath. So, when you when you focus on on the Unity Catalog, okay, we synchronize the metadata. You see that all this information you you find there is actually being able to
[28:45] be to bring back in the data intelligence platform, okay? And you see that we actually provide a lot more information like data lineage, business glossary, etc. back into into Unity.
[29:01] So, we are a perfect companion to Unity if you're using it. And when it comes to data observability, we do the same thing. We're able to measure and in this scenario Delta Lake and and
[29:18] bring it as well to the data intelligence platform or to Unity catalog. Okay. And uh when we we did that uh we we did that in in uh we with with one of our customers
[29:35] um Lufthansa Cargo. And you know, they're they're Germans, so they're really rigorous. Uh Okay, I'm going to stop the stereotypes, but uh and so, they had they had a bunch of XML documents that they were transforming
[29:51] using using using Databricks, and nobody laughs at XML. I love XML. Um and they just wanted to push it into into the data intelligence platform, and so, they actually could easily facilitate and and discover the
[30:07] data for their for their users. So, to uh to to wrap up, um
[30:24] for for easily four points. The first one is really, you know, data contracts are more important than you think. I The The first time I, you know, I worked with people to help me with with with the wording on things. And my first version of that was data contracts are important, which is kind
[30:39] of boring. But I said that Okay, but on the other hand, data contracts are more important than you think. Sounds a little pretentious, right? Saying that something like hey, I know better than you. Well, so I I had a very entertained discussion with my coach on
[30:56] on the topic there. And really he said that are you not thinking that it is true? Are you not thinking that is more important than people most people think? And I and but think about the field implementation that people actually
[31:12] did with data contracts. And it's true that after their implementation, after they saw the ROI, it became more important than they originally thought. So I kept the sentence.
[31:29] So really you get value extremely quickly with data contracts. Uh and then you can go to data products and then you can make your AI a little bit uh quite a bit uh um more performative.
[31:46] Open standards. I mean, we we we are at a conference that was built on the success of open source, right? It's built on the success of Spark. So I I don't have to justify the importance of open source. But open standards are even more
[32:02] important than open source. Because they allow this interoperability that we really need. We're thinking about let's just think like back in the '90s, if HTML would have been proprietary from
[32:17] from vendor to vendor. So web would not be there, okay? So I I I often like to compare OCS really to to HTML in this in this in this era of uh open standards.
[32:33] And we as a company are dedicated to open standards. And you, if you're working for an end user, your mission is to push vendors towards open standards. Because they don't like it.
[32:51] Honestly, they don't like it. Vendors, they want vendor lock-in, okay? So, why would they push open standards? But, it's your responsibility to push your vendors towards open standards.
[33:07] So, uh My My name is Jean-Georges Perrin, JGP. Uh please connect on LinkedIn if you if if if we're not. Uh the other things are I think it's Medium and Substack where I publish things as well. Uh so, feel free
[33:23] to to join those as well and uh continue the discussion there. And if you want to experiment with the workbench, well, it's it's workbench.actionlabs.com. Uh because the the QR which is probably not readable. Uh so, thank you thank you very much. And
[33:40] uh yeah, come come come meet us at booth 330. Uh sorry for not super readable, but 330. It's very close to the main Databricks booth. And uh I I'm
[33:55] I'm I would be more than happy to share my boss's book for free. And she even signed it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.