Skip to main content

Metadata-Driven Streaming Ingestion Using DLT, Azure Event Hubs and a Schema Registry

Summary

  • Real-time data ingestion: Plexure transitioned from batch to near-real-time streaming using Azure Event Hubs and DLT, processing hundreds of millions of customer activities daily.
  • Schema-driven automation: A Schema Registry enables data producers to evolve schemas with confidence and seamlessly publish new datasets without requiring Data Engineering assistance.
  • Operational impact: The metadata-driven framework accelerates time to insight, enabling real-time reactions to customer behavior for personalization engines and marketing campaign optimization.

Metadata-Driven Streaming Ingestion Using DLT, Azure Event Hubs and a Schema Registry

Watch: Metadata-Driven Streaming Ingestion Using DLT, Azure Event Hubs and a Schema Registry
Talk by Vicky Avison, Principal Data Engineer, Plexure
At Plexure, we ingest hundreds of millions of customer activities and transactions into our data platform every day, fuelling our personalisation engine and providing insights into the effectiveness of marketing campaigns. We're on a journey to transition from infrequent batch ingestion to near real-time streaming using Azure Event Hubs and DLT. This transformation will allow us to react to customer behaviour as it happens, rather than hours or even days later. It also enables us to move faster in other ways. By leveraging a Schema Registry, we've created a metadata-driven framework that allows data producers to: Evolve schemas with confidence, ensuring downstream processes continue running smoothly. Seamlessly publish new datasets into the data platform without requiring Data Engineering assistance. Join us to learn more about our journey and see how we're implementing this with DLT meta-programming - including a live demo of the end-to-end process!
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

How much customer data does Plexure ingest?

Plexure ingests hundreds of millions of customer activities and transactions into their data platform every day, fueling personalization engines and providing insights into marketing campaign effectiveness.

How does Schema Registry enable data democratization?

The Schema Registry allows data producers to evolve schemas confidently and publish new datasets seamlessly without Data Engineering assistance, reducing bottlenecks and enabling faster data platform growth.

What is the benefit of transitioning to streaming?

Moving from batch to near-real-time streaming enables Plexure to react to customer behavior as it happens rather than hours or days later, dramatically improving personalization and marketing campaign performance.

How does DLT meta-programming simplify data pipelines?

DLT meta-programming leverages metadata from the Schema Registry to automatically generate and manage data pipeline transformations, reducing manual code and enabling scalable data ingestion.

Full transcript

0:00 Introduction to Plexure

Okay. Hi everyone. Um, thank you all for coming. I'm the principal data engineer at Plexure. And today I'm going to be talking about how we've been overhauling the way we bring data into our data platform using this lovely mix of technologies. I didn't get the memo about the rebrand of DT to declarative pipelines, but just read read declarative pipelines. Cool. Um, so a bit of background at Plexure, we ingest hundreds of millions of customer activities and transactions every single day.
Uh, so a bit about what we do. Um I'm assuming most people don't know what anything about what Plexer do apart from the two people in the room who work for Plexer and I hope they know. Uh so we work with a very large multinational organization um fast food brand to provide their mobile application in 60 countries around the world. And what this means is in the app there's targeted offers, adverts, loss programs, things like that.
So we're kind of capturing the whole of that customer journey from opening the app, looking at an offer, going to a store, redeeming the offer. So this is generating a lot of data and then this data goes into our personaliz personalization engines which uh we use to recommend things to the customer, different targeted offers. So they get more things that they want and then they engage more with the business which makes the business happy.
So win-win. And we also do reporting on how offers are performing so that marketers who run the campaigns can make adjustments so they're performing as they want. Uh so as I said we're ingesting a lot of data and we want to do this faster. So what slows us down? Um so the really obvious thing is we're just not bringing data into the platform fast enough. We've got a lot of kind of infrequent batch ingestion pipelines.

1:56 Identifying pipeline issues

A lot of its legacy. Plexure grew very quickly and like many organizations that grow very quickly, there's a lot of kind of legacy stuff out there that wasn't necessarily built with the future in mind. Uh so a lot of this stuff isn't really able to run more often than once a day or twice a day, that kind of thing. So one of the things we want to do is move to much closer to real time. And then the other two things, this is one of them. um and less about the data speed coming in and more operational like how we operate as a business.
So one of the problems we have is these pipelines are quite brittle in terms of if we make changes to the schema the fields that are coming in uh it can cause issues. So just seemingly innocuous things to some people like changing the data type of a column somewhere can cause lots of things to blow up downstream. So just making changes is a bit of a minefield. And then the final thing is um I don't know whether anyone else has this problem but we find well we have found historically that bringing in new data sets to the platform is actually like a significant piece of work.
There's people on one end, people on another end, people in the middle, there's a project and it just takes quite a lot of effort to make it happen. So one of the things we were looking to tackle is just making that way easier and way more seamless. All right. So high level solution um we want to use a distributed streaming platform. So something like well we're going to use well we have used um as your event hubs but uh conflant cfka or just self-hosted cfka would work awesary serialization format that's good for scheme revolution.

3:23 High-level streaming solution

There's lots of support out there with tooling that works really well with it. Alongside that, a schema registry. So this is pretty key to the whole thing. This a schema registry is a service where you maintain your schemas for your well it works with multiple types of serialization but in this case AVO. So you maintain your AVO scheas in the schema registry. Schema registries are great for multiple reasons. one is that it provides a agnostic um way of interacting with schemas.
So for example, our people producing the data are net that's net applications and we're consuming it using uh Python DT in data bricks but schema registry is something in the middle that is agnostic to both of those things. So they can both talk the same language and it's also friendly for humans. So I could go into the schema registry and I can see the schemas that are going through the system. So it's a really good way of managing that multiple views of looking at the schemas and it also schema registries generally if you're using one of the common implementations they will do compatibility checks for you.
So if you make a breaking change to a schema something like changing a data type in a way that's not going to be compatible it will not let you do that. And then final bit um yeah declarative pipelines as they are now called this is what we're using to ingest the data into the platform. All right so the flow kind of goes a bit like this. Uh we've got our producers we've got net teams developing applications which will produce data to the event hub.
They will be developing their code. So imagine they've been told they need to start producing sales data to the event hub. They'll be working away and go right this is ready for production. This will then go into the C cicd process. Uh that CI/CD process will publish the schema of those events to the schema registry. Um assuming that all went well, the CI/CD process will continue and deploy that code to production where it'll start producing events to our event hub.
Um the messages that go on this event hub contain the serialized AVO data uh and along with the schema ID for that schema. Uh the schema ID uniquely identifies a schema in the registry. So a certain schema can have multiple versions. That schema ID uniquely identifies that version of the schema. And then on the other side, our DT pipeline will pick that up, get the right schema, des serialize, and then we'll write to a table in Unity catalog.
Uh, and that's the happy path. The unhappy path is if they then want to make a change to the sales schema and they change for example a string column to an integer, they'll be developing, they'll go, it's ready. CI/CD process will run try and register a schema in the schema registry that will reject that schema for being incompatible. CI/CD process will fail. That code never gets to production. So, we're massively safeguarding ourselves against problems ever making it to production by using the schema registry at that point.
Uh for the rest of the talk I'm actually going to be talking about the Azure schema registry which doesn't get a lot of love and lots of people don't even know it exists but I wanted to briefly talk about the confluence schema registry which many more people have heard about. There is a really nice integration in data bricks for working with confluent. Um so there is for those of you who don't know there's a function in open source spark called from avo that you can use to do der des serialization of avo but you need to know your schema up front so you pass that schema directly to the from afro function and I'll show that in a subsequent slide but with in data bricks there is a version of this function where you can pass it schema registry details for confluent and then it will go and basically handle everything for you.

7:15 Schema evolution and registry

So your schema registry options look something like this. Um that third line in the options uh a schema evolution mode restart this will basically make schema evolution works. So what happens is when your pipeline is initialized for the well when it first starts the data frame here will have the schema of the latest averro schema in the registry and subsequently your table that it's writing to will have that schema and this is all good that's exactly what we want but if this is a streaming pipeline it's running continuously and while it's running a new schema gets registered to the registry it won't like automatically pick that up and evolve the schema in your pipeline because it's running, it needs to restart for that to take effect.
Uh so what this property does, the restart property is when it encounters data that has a schema it hasn't seen before, it'll throw an exception, the job will fail and you can have it configured to restart. For anyone that's worked with autoloader, u it's a very similar thing with schema revolution. If it detects something new, throws an error, restarts. So the reason I bring this up is there is no lovely from a function for the Azure schema registry but we were really keen to get it working and this was kind of this pattern the pattern of it failing when it encounters new schemas and all of that was what we were trying to replicate in our setup.
So I'm going to do a deep dive. Uh disclaimer there's going to be a lot of code uh but then it'll be followed up with a demo. So if code's not your vibe for demos are, then hang tight. All right. So taking scheme registries out of the equation to begin with, we just look at the from a function and what that does. So this is what an a schema looks like if you haven't seen one before. Uh and then we can just use from a we pass the schema into that and then it des serializes.

9:32 Deep dive into deserialization

That's all good. That's what it works at bas how it works at a basic level. But then obviously we want to work with multiple schemas. So if we've got a version one of the schema here and then a v2 where we've added a new field uh we can create a little dictionary of version two schema and then we can work with it from there. So this is where I should probably pause for a bit and explain how from avo works with different versions of the schema because it does do schema evolution but it's a bit kind of tricky.
You can't just pass it the latest version of the schema and it just works. In a there's a concept of a writer schema and a reader schema. So the writer schema is the schema that it was written with that it was serialized with and then reader schema is whatever schema that's compatible with that you want to read it with. Um so in the context of this our reader schema would be what we want the we would want that to be the latest schema and that would be what the schema of our data frame and the schema of our table ultimately but we would always need to pass the exact schema it was written with to be able to des serialize it.
So that's the context for what's coming up. The way we making this work in DT is we're using the flows API. If you've used a pen flow before, that's what that is. Uh, so we first need to create our streaming table and then we do a bit of DT meta programming. So we loop through our schemas that we've defined and we call DT. append flow and we're writing to our table that we just created. So what we're doing here, each of those append flows is for a different version of the schema.
So we're filtering assuming we've got a column called schema version in our data. We're filtering that for each version of the schema creating a flow. They're all writing to the same table. And then the key thing is so in our from averro function we're passing the writer schema which is the one we need to des serialize but we're also passing this extra option called avo schema which is confusing confusingly named but we're passing the latest version of the schema for that.
So all of our flows will end up with the same schema, which is that of the latest version, but they all are deserializing with the one they were written with. This is the like hardest bit of all of this. Everything else just builds on top of that. Sorry. That was loud. Um, right, the Azure schema registry. So this is where it gets interesting because obviously we don't hardcode all of our schemas in our code. That wouldn't make any sense.

12:29 Azure schema registry integration

So enter the Azure schema registry. This is what it looks like. It's It looks like what it looks like. Um I'll show you in the demo later as well, but within the schema registry, there's concepts of schema groups. So, this is where you group like related schemas together and then each of those schemas has a name and then each of those schemas has multiple versions. Uh yes, that is what it looks like. All right. Then we need to get the schemas from the scheme registry.
So instead of that bit where we've got a hard-coded schemers, we need to actually go to the registry and fetch them down. There is a Python SDK for this, but it's a bit limited. It doesn't actually expose all of the REST API calls, and we need more than is available in that. So I've just created this little client extended. Uh we need to get all the schema versions for our schema and then we want to get the actual schemas.
So then we've effectively recreated what we had hardcoded. I'll just whiz through this. Uh yeah, then putting it all together. So this will look very similar to the version where we'd hardcoded our multiple versions of the schemas, but instead of hard- coding them, we're now getting them from the schema registry. Sorry again. Um right. So we first need to read our data from our event hub. So you might go, "Oh, you're reading from CFKA, not Event Hub."
If you haven't worked with EventHub before, it has this really lovely CFKA endpoint. So it just pretends it's CFKA and acts like CFKA, which is a really awesome feature of EventHub because it means anything that integrates with CFKA can integrate with it. So that's why it looks like I'm reading from CFKA. Uh, and then we need to get our schema ID out. So don't worry too much about this. But basically the schema ID is in the headers of the payload and kind of nested in this avo binary uh thing in the content type header.
Then we want to get our latest schema and then we're basically this is just the same code we've already seen. So we create our scheming table streaming table sorry loop through all our schemas append flow des serialize filter based on the schema ID. So this is exactly what we saw before with the hardcoded. We just have now have our scheas from the schema registry but we don't have the restart thing. So how do we make the restart thing happen?

15:06 Handling schema evolution

And this is we get all our schema ids that we've seen and we can add a DT expectation um that if we haven't seen that scheme ID before fail. So this is how we've kind of hacked the restart functionality. All right. So the next problem is how do we extend this to make it like automatically handle new data sets? That problem number three I talked about earlier where we want to make it easier to ingest new data sets. Our vision was, oh, we'd love producers just to start sending us new data sets and to handle that automatically.

15:28 Automating new dataset ingestion

So, no more big projects. And it turned out once we've done all that work to make the scheme revolution thing working, it was actually quite easy to extend it to this use case as well. So, this is kind of the picture we're talking about. So, we've kind of got producer A, producer B, they're happily already sending data to our event hub. What if producer C comes along? it just starts sending data and then our DLT pipeline automatically picks it up and creates a new uh table for that in Unity catalog.
So this is what we're trying to achieve. All right, so we just need to make a few changes. So instead of just fetching all the schema versions for a single schema, we're now fetching all of the schemas within a schema group. So we just need a couple more functions. So we first get all the schema names available in the schema group and then we get all of the schemas for all of them. Okay. And then we need to make a few more changes to our pipeline code.

16:20 Advanced DLT meta-programming

So we're getting all the schemas again. We just do a bit of structure manipulation. Sorry, I'm whipping through this a bit because I know people don't like to just stare at code for ages. Uh and then the main change here is instead of just having our one streaming table, we now create one for every single schema name that we have. And then this is exactly the same as it was pretty much. Um I think we maybe moved some things around.
I think later schema is now nested within the function, but it's basically the same thing. we're just writing to the specific table name that we need to. So yeah, this is the crux of it really. Create the streaming table for each stream uh schema name and then yeah do our append flow metaromming demo. Yay. So yeah. Uh right, this is where it gets exciting. um praying to the demo gods that everything is going to work nicely.

17:41 Live demonstration

All right, so I've got my little demo notebook set up. So this is our scenario that we're trying to show. My pipeline's running. So it's first ingesting just the raw data from the event hub. This is not deserialized. This is just the um yeah the binary data uh and all the event hub metadata as well. Right, I'll get running. I just need to do some initialization installs. Um I'm setting up my schema registry client. Uh this these are um Azure provided libraries.
Afro encoder again this is provided by Azure for encoding using a schema registry and the event hub producer. So this here is my basically this is my data producer notebook. All right. So the first thing we're going to do is register a new schema. And this is sales data that looks a bit like this. Cool. So I'm just going to run this. I'll show you the uh schema registry first. Just I don't know why I need to do this.
I just feel like you know in magic tricks we're like oh look it's perfectly normal deck of cards. Yeah. There's nothing here. Uh, right. All right. So, I've sent my schema to the schema registry. So, this would be like that CI/CD step where it's doing the registering. All right. So, we've got our schema here. We can see it in here. I could probably see it a bit better if I open it up. Yeah. All right. Fun fact, you can document your AVO schema.
This will be relevant very soon, but it's a really nice like nice feature within AVO. Cool. There it is. Still not doing anything because we haven't sent any data. And this is where I hope and pray that everything works as expected. Cool. Right. We're going to start sending some records. So, we're here. We're in the producer. We're going to get the schema from schema registry and we're going to start sending those events to the event hub.
So we're using the Azure libraries for encoding and sunning. I'm just going to send four sales records to the event hub. Now what we're expecting to happen here is the expectation will fail because it hasn't seen this schema before. It takes kind of an annoyingly long time for this to happen, which I wouldn't mind normally, but when you're doing a live demo, it's just a bit yeah, anxietyinducing because you're wondering whether it's going to work.
So fingers crossed in several seconds time, this expectation will fail and we will be able to see what happens. Yay. Okay, that's a relief. All right, so we can have a look at the expectation. I don't know how well you can see on those screens. Um, but we basically says, yeah, failed to meet the expectation, violated expectations, known schema, and then some more information. Classic expectation stuff. Sorry, I'll just put this away because it's taking the screen.
All right, so this should be now restarting. If anyone's done continuous pipelines with expectations before, you might know that I'm just preempting random questions. You might know that it doesn't actually automatically fail. Uh it kind of just keeps running even though the data set has failed the expectation. Uh I have a little watcher thing that's looking out for the pipeline events for when the expectation fails.
So I then restart the pipeline. Anyway, look, happy days. It's worked. So we now have our sales data coming through in our pipeline. Cool. Um, the next thing you can try to evolve the schema in an incompatible way. So, what I'm going to try and do here is yeah, change one of the data types. I think this was a string to an integer. So, I'm just going to try and register that in the registry. All right, we got an error.
Yeah, a schema. Yeah, don't try and help me. Aro schema validation failed. Backward compatibility test failed when checking against schema sales version one. So this is what we want to happen. This is our CI/CD process going no you can't do that. We're not putting that code into production. All right. Now we want to evolve the schema in a compatible way which is just adding a new field. So if we create a field called new field uh it can be null.
Its default value is null. And then we add it in. Yeah. There we go. We can add that new schema to the registry now. in the registry, right? We now have two versions of the schema. So version two has this new field in it. Again, this hasn't done anything because we haven't actually sent any data, but then we shall send some. All right. So, I've sent two sales records with the new schema to the event hub. So, now there's a bit of a wait for it to fail again.
Uh, in the meantime, we can have a look at Unity catalog. So, our table is in the catalog here. All right. This is a little fun bonus that I didn't talk about in the presentation because I thought it was already quite a lot of slides, but we've got kind of auto what I call auto documentation. So, if you remember from our schema that's registered in the registry. We've got these little dock. There's a dock on the record itself.
This is kind of like the dock for the whole event. And then we've got little docks on each of the fields. So what I've done is taken that and made that go into Unity catalog as documentation for our data. This is really cool for us because the people producing the data are the ones that really know what that data is all about. So they're the best people to document it and then that way they're documenting at the source and then we get it and it's in Unity catalog and it's great.
Um I can show you how we do this. It's just bit of utility. So if we go here is my pipeline code. Um this is literally what was in the demo. So it should hopefully look familiar. But this is the magic. Uh so when we're creating the stream go sorry uh when we're creating the streaming table we can pass a comment which is the table comment and can specify a schema and this schema this is a strct type but you can specify comment data comment metadata in that strct type.
So I've got a couple of little utility functions for this. Um we go in here. Oh look more code from the demo. It's real. Uh yeah get table comment. So just getting the dock out of the a schema. Getting the uh the column comments is a bit more complex. We need to create a strct type based on the AVO schema. The easiest way I could find to do this was just to kind of use Spark and kind of read format a nothing. uh but you get a schema out of it and then get the comments from the a schema and then yeah add these as metadata to our strruct uh type and then return that.
So this is what's going into here. Uh so that's how the auto documentation is working which is pretty cool. Anyway, this restarted we got our new evolved sales coming through. Then we want the interesting bit which is registering a new schema. So, we've got some click stream events this time, right? I'm going to register a new schema in the registry. This is a completely different data type now. Not a different version of sales.
It's just a completely new one. And then I'll start sending some events. All right. So while we're waiting can have another look at the schema registry. Yeah. So we've got our new event type here. Kind of looks the same. Sweet. So now we've got two data sets coming off. So now we're automatically producing two different tables. Yeah, we've got our click stream events coming through nicely. So one of the things I can show I can start sending some other sales.
I can also send the old version of the sales uh and that'll work fine. Uh, so this is the original schema. So we'll read with this is where the reader writer schema comes in here. We'll be reading with the latest schema, but we'll actually be deserializing with the one it was written with. Right. So we sent some more events. They should just come through. Yeah, look six popped up here. Yeah, we've got some records here.
So we've got everything going to the right place, which is fun. Uh, have another look at our tables. And we now have yeah our click stream events here as well. Start having a look at what's in here. Yeah, we sent two records here. Sales. We've obviously got quite a bit more. And how I see over there some reason I don't seem to be able to see. Oh, there we are. So, new fields. So, we only actually sent two that have that.
Um, four of these predate when we evolved the schema and then I just sent four more uh with the old schema again. So that's why they're all coming through with nulls. Um, and if you're interested, we can have a look at the raw data which doesn't really tell us anything. But this is the magic. This is the value. This is our afro binary serialized record. And then there's some extra metadata here and the scheme ready. Cool.
So that is it for the demo. Um, I'll just go back here. All right. So, the key thing is, have we actually solved our problems? So, yes, no more infrequent batch ingestion. This can be done near real time. Well, the point is it's up to us now. So, we if it doesn't need to be near real time, it can be as often as we need it to be, but it can be near real time. Um, evolving schemas is risky. Our schema registry has helped us with that because we've now got guarantees when a new schema is being registered that it'll be rejected if it's not compatible and the CI/CD process the producer will fail and it will never make it to production.

29:03 Conclusion and Q&A

Happy days. And bringing new data is significant piece of work. No, because producers can just start sending new records to the event hub and we'll automatically pick it up and create tables within data bricks. So it's very exciting and that is it for my presentation. Thanks very much. I'll be happy to take questions. I think we do have some time. So if there are any happy to So, thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.