From Factory Floor to AI-Ready: Building Industrial Data Pipelines for Databricks

Summary

HighByte addresses the missing context layer in industrial AI by contextualizing and modeling OT data—from PLCs, SCADA, historians, and MES systems—at the source, producing AI-ready data products that arrive at the Bronze layer ready for Databricks medallion processing.
The HighByte approach covers four industrial data source categories, handles semi-structured and image data in Unity Catalog volumes, and uses schema projection to dynamically onboard new assets without manual pipeline changes.
A Gousto case study demonstrates the results: 98% equipment availability, a 10% throughput gain, and 5x more recipes achieved by combining HighByte, AWS, and Databricks for predictive maintenance and digital twins.

From Factory Floor to AI-Ready: Building Industrial Data Pipelines for Databricks

Industrial data is unlike other data. Industrial companies across manufacturing, energy, life
sciences, and logistics run on PLCs, SCADA, historians, MES, inspection systems, AGVs, and more.
This data is more raw and harder to work with than IT data. Standard ELT pipelines move this
problem downstream rather than solving it. As industrial companies race to adopt AI, this
problem is amplified further.
Industrial DataOps addresses this by contextualizing and modeling data at the source, producing
AI-ready data products that arrive at Bronze ready to progress through the medallion
architecture.
Gousto, a UK meal-kit manufacturer, put this into practice with HighByte, AWS, and Databricks.
Their Unified Namespace and Lakehouse became the foundation for automated fault detection,
ML-driven predictive maintenance, and digital twins, driving 98% equipment availability, a 10%
throughput gain, and 5x more recipes.
Learn how to unlock the value of your industrial data with Databricks.
Talk By: Jeffrey Schroeder, Director of Product Management, HighByte ;

Chapters

FAQs

What problem does HighByte solve for industrial companies using Databricks?

Industrial data from PLCs, SCADA systems, historians, and MES is far more raw and semantically fragmented than IT data, making it very hard to use directly in lakehouse architectures. HighByte contextualizes and models this data at the source—resolving naming conflicts, normalizing schemas, and enriching it with asset context—so it arrives at Databricks' Bronze layer already AI-ready.

What is the E-stop problem and why does it matter for industrial AI?

The E-stop problem refers to the same physical event—an emergency stop—being described differently by different OEM devices, with conflicting labels, units, or schema structures. Without a contextualization layer like HighByte, these conflicts propagate into analytics and AI models, making outputs unreliable across the fleet.

How does HighByte integrate with Databricks Unity Catalog and Genie?

HighByte can store semi-structured and image data in Unity Catalog volumes, uses schema projection for dynamic asset onboarding, and exposes historian data to Databricks Genie and AI agents via MCP tools. It also supports reverse ETL, pushing AI-ready insights back to the factory floor for operational use.

What results did Gousto achieve with HighByte and Databricks?

Gousto, a UK meal-kit manufacturer, used HighByte with AWS and Databricks to build a Unified Namespace and lakehouse foundation that enabled automated fault detection, ML-driven predictive maintenance, and digital twins. The outcomes were 98% equipment availability, a 10% throughput gain, and 5x more recipes produced.

Full transcript

[00:07] Uh so today we're going to be discussing uh industrial data in context. Uh first let's level set on has has everyone familiar with what medallion architecture is? Have you heard it before? So everyone seems to be nodding right you kind of load data in you
[00:19] transform it you make it available or you create KPIs on top of that. One thing I've noticed a lot the past few days at the conference is the amount of times the word context has come up. It's coming up a lot. Um, uh, it's almost
[00:31] like there's a layer that's missing, if you will. It seems like agents have kind of amplified a lot of problems that we've long had, um, with our own data. In some aspects, this is not really all that different, right? Um when you think about business intelligence, think of
[00:44] how many times uh you know users have in the presentation layer have had uh maybe um KPIs that are cross-sectioned by maybe BEu or region or over time periods and they suddenly want the total or the
[00:56] aggregate and they take the average of all the quarters of their profitability or they take the average of of their business units or by line and so forth. These problems have existed for for a while, but it seems like there's this problem with there's almost a missing
[01:08] layer on the other on uh at the top end and agents are just making the problem even worse. But we're going to focus on today is on the other end of the spectrum where there's a there's almost missing layers from a contents perspective. So typically when people
[01:20] use uh lakehouse architectures, they're often taking uh it data and they're loading it into bronze. And there's certain assumptions about uh um about IT data, right? Generally, it's tabular. It's coming from applications, it's
[01:32] coming from databases, if you will. But when you go into the industrial domain, the data is significantly more raw. And when we talk to when I personally talk to a lot of manufacturers or industrial organizations, often they seem to be struggling by saying, "Hey, I'm moving
[01:45] data in there, but it's too hard to process. I have all these challenges, if you will." So, it's almost like these like there's some sort of layer missing. And so, I sometimes almost like to joke is we're trying to move data that isn't bronze into bronze. We're trying to move
[01:57] something far more raw. So today we're going to focus on this kind of industrial ETL aspect to to working with our industrial data. So why is industrial data a mess to use? Uh first of all, anytime that you introduce nodes
[02:10] in uh um in an industrial environment, that is the things that produce data um they're often stood up for a very very narrow purpose. The most obvious thing, right, is that their long life cycle, right? So these things we get a new one
[02:22] about every one to two years, every 3 to five years. If you're a manufacturer, uh it's not uncommon to have assets stick around for 20, 30, 40 years. But the more but the bigger problem it seems to be is that when people implement things because they're so specialized, uh the
[02:35] the their interoperability, their data requirements and so forth are very very distant criteria, right? And these things are just kind of sticky, right? And so in many cases, right, um uh uh meeting a deadline to get a line up and
[02:47] running, to get a line install complete, to do a changeover and so forth, right? Those are far more important concerns than than any sort of interoperability. So architecture tends to be a mess. The other issue that you'll also see in the industrial domain is just the amount of
[02:59] data variety, right? A lot of people fixate on on telemetry data. So like tags and so forth out of industrial control systems, but there's actually a lot of transactional systems in there, databases if you will. If you're a batch or or process industry uh um uh type
[03:12] manufacturer or industry, right? Uh you typically may already be collecting data in time series databases. And the other thing that's really really interesting is the amount of data that's locked up in files. I can't tell you how many times I speak to manufacturers where yes, they've tapped into their tags, but
[03:25] their whole quality lab or some of their highest value data are stuck in images or structured text files or or semi-structured data structures. And probably the most painful aspect is the missing context. Right? Each of these systems were set up for a very narrow
[03:38] purpose. Enough to just produce widgets or enough to support the production of widgets, but they don't really understand how they relate to other nodes. uh they're designed for a narrow narrow purpose. And then finally, the biggest component that's that's a real
[03:50] struggle is just there's not enough resources, right? Uh yes, some of these use cases have value uh but ultimately you just you just run out of resources. So I'm going to quickly just run through some scenarios of like what what are
[04:02] challenges or what is what is necessary to make interoperability work in the industrial domain. So the first thing is a lot of people just focus on converting protocols, right? And that's that's that's great and so forth. But if you look at the second scenario there,
[04:14] something as simple as an estop button, even after I convert everything to one protocol as you see there, uh I'm describing the same type of phenomenon or the same event, something different, right? So if I'm just streaming that uh desperately into data bricks, right?
[04:26] That's going to be a nightmare, right? I'm not solving the problem, I'm moving the problem, right? The other aspect is uh industrial data sources rarely have the full context. People keep you overusing the word context, but but it's a a really huge problem there. meaning
[04:38] that when you look at it like let's say a CNC machine or industrial asset, it doesn't know what material it's running, it doesn't know what operator that's on. It's just a basic computer that's running a schedule over and over again to read sensors and act and fire off
[04:51] actuators over and over again. A lot of the context that you need often lives in other systems. So, it's very common that you have to be able to join data set sets together. There's also issues with velocity and and being able to model data to the particular needs of the
[05:03] consumer. So, I'm going to double click on this whole context aspect. If you look at that data set on the right hand side, um that's a pretty good data set, right? Uh that would be ideal to sit in a nice
[05:15] wide table in in your data bricks environment. That would be good for AI, right? But if you look at let's say that attribute power pump consumption 35 and we go ahead and look at the data access pane on the lefth hand side there and we see a register 40,13 that's what data
[05:28] looks like in the field in the real terms. So the question though is if our data strategy is merely just loading data or moving it, you can quickly see that's going to be a disaster, right? We can quickly see that 40,35
[05:41] means nothing to anyone, right? And the problem is you don't just have one of those. You might have hundreds of thousands if not millions of these within a single facility. And the other aspect is that you might the context might be spread across multiple data sources. So it's not just a matter of
[05:53] moving the data, but we actually need a pipeline and join data sets together. So I work for a company called Hibite. We're an industrial technology company. Um, we've been in the space for a long time indirectly with previous companies. I personally come from the automotive
[06:06] industry. I have a lot of colleagues that have been working on uh industrial software for many years. In some cases, you probably have been using it for many many years. So, we work in a lot of different verticals and so forth, but really we're we're focused on we only focus on industrial data. So, we offer
[06:20] software called uh for industrial data ops. It's referred to as the hibite intelligence hub formally. But I don't want to make this so much of a commercial, but what I really want to talk about is what are the four key pillars that we need for interoperability, right? And how what's
[06:32] our assumption on industrial data ops? So the first is data orchestration. Yes, it's important to have connectors, but it's more importantly to have uh pipelining and modeling capability, right? So I'm going to keep driving this point home. Don't move the problem,
[06:44] solve the problem. Right? You need to be able to orchestrate interactions between uh data uh um producers of data and consumers of data. We also need to be able to observe those data pipelines. We need to make sure that they're healthy. We need to make sure that uh uh that
[06:57] they're running, that we're not getting data loss. There's also unique requirements in industry for data quality, right? When we typically think of the IT realm, we're typically kind of frustrated on saying, "Oh my gosh, my order data was was entered incorrectly for a purchase order and I have all
[07:09] these data quality problems and I can't report against against that." That's not so much as big of a problem in the industrial domain, but it's just more about the heterogeneity. all these different types of assets and systems or if a network goes down, how do you safely recover from there? And finally,
[07:22] the last is is governance, right? How do you scale this stuff out? So, we got to break these up into four key pillars. We connect to a wide variety of systems and focus on the industrial data, but one of the most important nodes for our for the organizations that we work with is data
[07:34] bricks. So, we break up data from a source perspective into four key buckets, right? So, I think for every manufacturer, don't fixate strictly on the telemetry. Too many people get way caught up on their asset data and you
[07:47] talk to your line of business users and they say, "I'm not actually interested in asset data. What I actually want is how that asset was performing, how it produced the process." In other words, I want to look at asset with a process context or a product context. Right? They don't want tags. They want data
[08:00] products, if you will. There's also transactional systems. You might have specific execution systems running on your plant floor. You might need to be able to loop APIs and so forth. If you're a batch or process industry, you might already be collecting your data
[08:12] into historian. That's very common in regulated industries like life sciences or or perhaps um uh oil and gas and so forth where they already have to log the data and a lot of the context is in there right and then we already talked
[08:24] about files but that's also a huge aspect of these things. So think less about the source and syncs of the data but think about the ability to be able to orchestrate and create interoperable pipelines. So there are multiple ways to
[08:36] integrate with data bricks and I think it's important to kind of break these up into a couple different modalities. So the first is bulk right so for a long time before data bricks had zero bus and all these newer technologies right we were generally loading data into object
[08:48] storage and and pulling data out of it so forth um there's a lot of excitement over the low latency ingest solutions but for a lot of manufacturers in many cases they've already collected their data and so why are we paying a premium or rushing data that is already two
[09:00] years old to get into data bricks right so sometimes the most efficient tool is to simply just use your classic object storage and use the autoloader the other aspect why this bulk method is particularly valuable is also data bricks as it's evolved its compute. It's
[09:12] not just about tabular data anymore. It also can work with images. So unstructured data, semi-structured data. So we can land data directly into the Unity catalog volumes and you can do your compute on top of that.
[09:24] So this is a great uh so it's a great approach if you already have lots of bulk data sets or you need to move unstructured data. Lots of excitement since last year about data bricks zerobus. Um for a long time you know people have been using uh um uh uh spark
[09:37] uh structured streaming to stream data in. uh we just shipped a data bus uh uh sorry a zerobus connector um in there that's great for low latency ingest but the other thing that we've also found that's a challenge with a lot of is that it's an appendon API right and so what a
[09:49] lot of people want to do is they want to control the schema and so we've seen contention where let's say you have assets right and you're bringing on new use cases or you're moving things on say you introduce a new asset class like let's say you're a new site and you suddenly have an injection molding machine or a robot right and maybe
[10:02] you're landing data in wide table formats in in data bricks right it's going to be kind of a nightmare to wait for someone to decide how am I going map robots into my data pipelines and then create tables inside of there. So we kind of have this concept of projecting schema, right? So that is as new assets
[10:15] show up. Uh it's important to be able to dynamically create data, land it into the bronze layer for ingestion. That takes a lot of friction off. Otherwise you have to coordinate three different teams in many cases. Sometimes that scares people. Sometimes a lot of people are say that's great. That's a lot less
[10:28] headache and that means that we have less less uh a lot less of a a backlog of data pipelines. And finally, uh, data within data bricks does not have to merely be stranded in the, uh, into the in the gold layer, the business intelligence layer. We hear from a lot
[10:41] of users where they're saying, I don't want my frontline workers to have to swivel chair or pivot between PowerBI and uh, their SCADA screen, right? They might want to take those insights and embed them directly in their operational tools. So, we also feel that's very
[10:53] important to have that reverse ETL capability. So, be able to do cloud to edge, call your data products or master data and bring it back down to the edge for for use within operations. And finally the um the last emerging pattern that we've quite seen is data bricks is
[11:05] has gotten very very good in terms of the lakehouse the the governance is all there the infrastructure sorted out so it's a really natural place to build agents one of the emerging patterns that we've seen from a lot of organizations is increasingly they're now building uh
[11:17] they want to do agents directly in data bricks but not just with data that's in the lakehouse but external uh data sets right so it's almost like you do analytical agents on top of the lakehouse but you might have operational systems so that is you have agents with
[11:29] shorter decision loops that you might want to use data. And this is particularly advantageous for organizations that have not figured out their lakehouse strategy yet. We sometimes hear from users where they're saying, "Well, it's going to be six months until we figure out how we want to replicate the data, how we're going
[11:41] to build our pipelines. Is there something that we can do now? We've already collected 20 years worth of data on our on-remise historians. Is there a way that we could tap into that now?" Right? And so what's a great approach is you could build mc custom MCP tools, expose that to Genie and get started and
[11:54] allow your users a quick win. So what I want to do is quickly talk about Gusto. So, Gust is a meal kit provider or a meal kit manufacturer based in the UK. In the US, we the probably the most well-known entity is
[12:06] is HelloFresh, uh, if you will, but if you're from the UK, you've probably you may have heard of Gust before. These are custom tomade, uh, uh, recipes and so forth, and they do they do deliveries, right? This company like blew up during the pandemic. They became quite
[12:18] literally a household name and so forth. So, um, this they're a very very sophisticated AWS shop. They had a lot of different needs for their for their data. And so we so we don't just deliver data to data bricks we deliver data to a
[12:30] lot of systems. One thing I thought was very interesting about their journey was no matter where they were in their journey they were always working with data bricks to figure out where is my problem and measuring it if they successfully solved it. So if you look
[12:42] at a lot of those applications that you see to the right of data bricks there those are all predicated by using data bricks. So the idea is they found out that they were having uh scaling issues, right? They had low mechanical availability. They have issues with um
[12:55] with uh line stoppages and so forth, right? They had uh slow meanantime to repair. Their maintenance teams were taking almost an hour to be able to respond to issues, right? And so they weren't really utilizing their capacity intelligently, right? So they figured
[13:07] out, hey, looks like we should probably invest in a CMS, a maintenance software with with mobile devices and so forth. They kind of benchmarked that based off here's what how much this problem is costing us. Here's how we solved it. and data bricks is has kind of been their key observability agent and so forth. So
[13:20] a lot of different systems on there and that's why you probably see so many icons because they're solving a lot of problems. But the one that out of all their f their use cases, there's one that I really really want to zero in on. This is where where is how they built digital twins. So they're moving data
[13:33] into data bricks to be able to observe their process. But one of the main issues that they found was after they solved their maintenance issues, uh they they were able to absorb their capacity. they're able to uh um uh uh hit their 98% equipment availability. They're
[13:46] they're having all the um the throughput that they desire, but what they really want to do is respond to the market and say, "We don't want to limit our customers with the amount of meals that we're offering." In other words, they wanted to be able to to utilize all this capacity that they just spent time
[13:58] optimizing, but they wanted to deliver more products. Right? The problem is, if you look at all those containers right there, they they have a total of they need to ship 200,000 boxes per week. They need to pack one every three seconds, and each box has 50 ingredients. Now at their baseline
[14:11] they're starting with 40 unique products per week. They now want to do a 5x increase to 200. Can you imagine all those containers the logistic night the logistical nightmare of carrying all that stuff around there. So what they did is they delivered that data into
[14:23] data bricks and they worked with any logic and they built a digital twin on top of data bricks right and this allowed them to figure out what is the optimal routing pattern for all these packages and that allowed them to effectively scale their recipes that they deliver per week from 40 to 200
[14:36] right so that allowed them to customers they're going to buy more things. they're not sick of the same recipes. Uh uh there's more variety so they can capture more value of the market. What I love about this is this really gets to the heart of industry 4.0. So um
[14:50] in Germany the phrase industry 4.0 isn't strictly a marketing term like it often is in North America. It's actually an it's a matter of industrial policy. It's a research term, right? But one of the key objectives of industry 4.0 was saying it doesn't matter if you have the
[15:03] most sophisticated and efficient production techniques in the world. If you can't respond to the market fast enough, it's ultimately futile. And so what I love and so one of the key things that researchers have have been trying to figure out this the space for since the dawn of industry 4.0 is really um uh
[15:18] uh how do we break down this trade-off between scale and scope. So otherwise, how do we manufacture at or near a lot size of one? So like mass customization as many products as possible without destroying our unit economics, right? So, other words, how do we how do we um
[15:32] manufacture as if we have a scale of infinite, but we're only making one product, right? And so, there's always this inherent trade-off between the two. What's amazing about Gusto is they were able to basically break down this trade-off and effectively get more, right? So, they're now at 200 recipes,
[15:44] they have all their their effective throughput, right? And I think this is just a great story about how when uh you know, a great team is backed by a strong ecosystem. So um Gustto and like many of our reference customers here uh have seen a
[15:57] lot of benefits from industrial data ops but really a lot of it comes down to time to value right uh um it's about uh reducing that marginal cost of integration. So that is if you need to add a new use case if you need to add a new model if you add new assets if you
[16:09] add new applications that cost of change should be as close to zero as possible when you integrate on a linear scale uh you you eventually run out of out of out of juice. Basically, you might be able to um uh work on the highest value use
[16:21] cases, but eventually you start running out of steam and you either struggle to be able to take on medium value or lower value use cases, but you also quite frankly end up struggling with the higher value use cases just from a maintain from maint from a maintenance perspective, right? You might find that
[16:34] that people are uh moving assets around or you might switch your uh your technologies around then all these pipelines break, right? So, I want to quickly just wrap things up. So we kind of talked about medallion architecture at the beginning of the the um the uh
[16:47] the session. What I wanted to quickly land on is just there's this missing layer that that I sort of believe. So at the top as you saw during the keynote, data bricks is thinking a lot more about um about semantic layers about uh the um
[17:00] uh uh ontologies and so forth, right? And so that kind of makes sense that there's been a missing layer on there, but we've seen we believe that for many years there's been a missing layer below that bronze layer. And it's really two things. And this is the probably the biggest mistake that we see
[17:12] manufacturers. Too many people either move their raw data directly uh to their enterprise systems or the or what they try to do is they fixate too much on asset models. Again, your line of business users don't care about tags,
[17:24] they don't care about assets. Please ask someone in your business, is there a spot in SAP where assets go? No, in many cases that's not. So that belongs in a SCADA that belongs in a historian. But line of business users want to know what was my asset in its process context?
[17:36] What was my assets in its product context? what is my asset in my supply chain context and so forth. So we provide that kind of critical layer of of uh solving that kind of iron and copper copper area right so in other words you use data engineering with high
[17:49] bite uh that that effectively walks so that the data engineering and data bricks can run so I want to thank everyone for for joining it's been a great week I'm here with my colleagues um uh if you are curious about industrial data we have lots of
[18:01] educational material online um we're happy to um answer questions and uh keep the conversation going thank you so much

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.