SQM: From Fragmented Data Silos to Real-Time ML Operations with OpenSharing

Summary

SQM, a mining company, overcame fragmented operational data silos — including plant historians, lab systems, and simulation tools — by using AVEVA's CONNECT platform and Delta Sharing to stream industrial data directly into Databricks on Azure without custom ETL pipelines.
The integration unified minute-level operational data views across iodine and nitrate processing operations, enabling the development of machine learning and optimization models that reduced production cycle time by two hours.
Delta Sharing's zero-copy approach means operational data from AVEVA CONNECT flows into Databricks without data movement, preserving governance while giving ML engineers access to real-time plant telemetry, chemical lab results, and truck fleet telemetry.

SQM: From Fragmented Data Silos to Real-Time ML Operations with OpenSharing

SQM set out to increase production efficiency by improving how raw materials are converted into finished product. But, it faced a classic data engineering challenge: critical operational data was trapped in silos (plant historians, lab systems, simulation tools) with no scalable way to unify it with enterprise data in Databricks. Operating decisions relied on operator experience rather than predictive models, leading to excess raw material use and inconsistent output. Instead of building custom ETL pipelines to move data across systems, SQM leveraged OpenSharing to create a governed, real-time data backbone. AVEVA’s cloud-based industrial intelligence platform, CONNECT, seamlessly shared data with Databricks on Azure using zero-copy and no data movement. This enabled operational data to be unified and refined into consistent minute-level views, and used to develop machine learning and optimization models.
Talk By: glenn moffett, Solution Architect, AVEVA ;

Chapters

FAQs

What data challenge did SQM face before adopting this solution?

SQM's critical operational data was trapped in separate silos — plant historians, laboratory systems, and simulation tools — with no scalable way to unify it with enterprise data in Databricks. Operating decisions relied on operator experience rather than predictive models, leading to excess raw material use and inconsistent production output.

How does AVEVA CONNECT integrate with Databricks?

AVEVA's cloud-based industrial intelligence platform, CONNECT, uses Delta Sharing to share industrial data with Databricks on Azure using a zero-copy, no-data-movement approach. This allows operational data to appear directly in Databricks as governed tables without requiring custom ETL pipelines to move data between systems.

What types of operational data were unified in this solution?

The solution combined chemical LIMS (laboratory information management system) data for quality analysis, plant telemetry from AVEVA's operations platform, truck fleet telemetry, and simulation model outputs. These diverse data sources were contextualized into consistent minute-level views in Databricks for downstream ML model development.

What business outcomes did SQM achieve with this architecture?

By unifying operational data from multiple plant systems, SQM was able to build machine learning and optimization models for iodine and nitrate processing. A simulation optimization effort achieved a two-hour reduction in production cycle time, improving yield efficiency and reducing raw material waste.

Full transcript

[00:10] It looks like my my time has started, so I I should get going. Can anyone hear me? Cuz I can't Okay, got some got some okay, that's great. So, welcome everyone. Uh just to start off a a forward-looking statement from Databricks. I should also mention as well that uh during the presentation,
[00:23] we'll be doing some demonstrations. They won't be using the customer's live data, but the customer did implement the solution. But as I'm doing some of my demonstrations, I might be showing features of uh our product that aren't released yet and
[00:35] might change before release. Just a heads-up about that. And and also the events team does a lot of work, so I always request that you uh fill out the survey at the end. Uh so, welcome everyone. Uh today we're
[00:49] going to be talking about how Aveva's industrial platform and Databricks uh combine together combine together to enable SQM uh to increase nitrate uh yield. My name's Clem Moffet. I'm a
[01:03] solution architect at Aveva. I work on a team called Lighthouse. We work with customers that who have specific use cases and then with products that are new or just released or maybe beta. And
[01:15] we work together to implement the customer's use cases and we both learn from that. Hopefully the customer likes it and keeps using it. And that's an example of the story that we have today. One of my colleagues who worked with SQM on a Lighthouse program uh to try out
[01:28] the Delta sharing integration and they're continuing to use the product today. And that'll be the story that we talk about. Does anyone know about Aveva? I say Aveva. Anyone know? Okay, maybe two or three hands, that's
[01:41] good. So, maybe new to a lot of people. So, Aveva is an industrial intelligence platform. Uh they support the whole life cycle of uh a the industrial customer. So, on the
[01:53] left-hand side, you've got examples of 3D applications that enable customers to walk walk through their environments and design their environments. In the In the center there, you've got the idea of
[02:05] operations where we help customers do the day-to-day operations. You may have heard of products like PI or maybe Wonderware or Aviva MES that are used to that operate. And we have a number of of
[02:17] customers across different industries: power, chemicals, oil and gas, pharmaceutical, mining minerals of course that we're talking about today, and consumer packaged goods just to name a few of the industries. And then the
[02:29] third column, an area for Aviva is around optimization. They don't do it okay, well, I've implemented my how do I improve? So, we have simulation software and analytics type software packages as well.
[02:41] So, a little bit about SQM and I'm presenting on their behalf today because they weren't able to attend the event. They're a diversified chemical company. Uh they've been around for about 60 years. Uh they're in multiple
[02:53] industries. One of those is lithium. They've also received a number of accolades as well based on their innovation, security, and also around plant nutrition, especially in harsh
[03:06] environments. So, this lighthouse project that we did with the customer was focused on a facility they had in northern in northern Chile where they made two
[03:18] products: iodine and nitrates. So, they mine cal- caliche oil. Sorry, caliche caliche oil. Let me get that right. I'll say it again. They mine caliche oil and they go through a
[03:30] process of leaching and processing to make iodine that comes out They deliver in drums. You may be familiar with iodine. Maybe you've you've used it as a vitamin as part of a vitamin potentially. But then a byproduct of
[03:43] that process of the second step of the leaching is a nutrient-rich nitrate solution that they then process through various plants to make uh plant plant nutrition products. So, for
[03:56] example, fertilizers. And so, the focus of the lighthouse, they decided to pick one area and it was around this intermediate plant. And it's a plant that uh does uh does crystallization. So, crystallization of
[04:10] nitrate salts. And a key aspect of that plant to operate that process is temperature. And so, they wanted to to see about how could they improve and then increase the yield of the of the nitrates by being able to tune that uh
[04:24] temperature. So, historically, they had operators who would just change the temperatures manually. They decide, "Okay, we think we should change it this." But they wanted to kind of get away from that and
[04:36] move towards a more data-driven process. And so, to do that, they had to do a a couple of things. One, they had to bring some data sets uh together. So, they had all this operational data and they have a PI system. And they also had some
[04:49] chemical analyses coming from a LIMS system. They had to bring those two data sets uh together and then run a simulation on that data to get a predicted temperature value. And then get that back to uh to the operators.
[05:03] Another key aspect about this type of solution, and not necessarily unique to SQM, is the fact that data is messy, right? It's not always up-to-date or correct. So, a concern for them was when
[05:15] they're moving that data and getting it together, they want to make sure that it's the most up-to-date and recent version of that data. There can be issues with out-of-order data, late-arriving data, and also situations where you have to do some
[05:27] re-recalculations. So, there's a number of opportunities and challenges here. And of course, the overall goal of maximizing that nitrate. So, before we talk about the solution, one of the aspects of the solution was
[05:40] uh Connect. And Connect is AVEVA's industrial cloud platform. Provides a number of services. Customers and also partners can make their own applications. And we have a calculation
[05:53] and modeling engine, a UI-based models that are pre uh pre pre-defined. We have tools around visualization for ad hoc visualization, but also for application-based visualizations that
[06:05] users can create. And then of course, connectivity. Being able to get data in from one or more of several AVEVA products, but also third-party products as well. Standard protocols. And of course, one of the integrations today,
[06:17] the collaboration that we're talking about, which is uh the Delta Sharing as well. So, if you look at the solution architecture for this, we have a data flow starting on the left-hand side from
[06:29] all that operational data that's in in in the PI system coming from from their plant. With that number two there, we have an agent running on Windows that we install. And then from Connect, uh the item three there, that's where we
[06:42] configure what data we want to retrieve from that PI system. We then make that available uh as part of Delta Sharing. And then on the Databricks side, they can query Connect for the data because with Delta Sharing, there's no copying
[06:55] of data. And with the chemical LIMS data, they can combine that and then run that simulation model to get that temperature value. But that's not the end of the story. We need to get that temperature value back
[07:07] to the operator again. So, we have a process in in Databricks where they can write that data back into Connect. And similarly, there's an agent that can be configured to move that data back down to the operational PI system where the operator
[07:21] is actually working, point number eight there. A note about this slide, too. All the steps, except for five and optionally six, are configuration only, which is important because some of the operational teams may not be coders.
[07:35] They don't want to have to uh go in and make and make changes to enable this this uh architecture. And also, this architecture is a baseline going forward so that once they do this one use case
[07:48] they can use the same platform as it is for future use cases. So if you take take take take a few minutes now to go through some of the steps of how would you actually
[08:00] configure this and and get the data flow through and get those temperature temperature results back. Uh we start with uh the the data in the Aveva PI system. Uh this is a platform that enables us to ingress all that
[08:12] industrial data be it from one or more or numerous pro protocols. But quite often when we bring this data in it's not very named well it's not organized well and so while someone on the plant
[08:25] might know what this means someone who is working as a data scientist has no idea what this tag name or the name is. So Ali this morning in the keynote talked about that contextualization. So in the PI system we can do the
[08:37] contextualization and you're seeing that here that instead of some tag name that that might be hard to decipher could just be a number now we have human readable ideally names that that
[08:49] describe that that describe aspects of the operation be it raw data be it calculations or metadata as well. And also on the very left we have a bit of a hierarchy as well the idea being able to create relationships between your data.
[09:03] Also if you're looking at this you might notice it's truck data it's not the SQM customer's data we couldn't demo the customer customer data so what we'll be using truck data today just for the demonstration purposes. So once we've got that contextual rich
[09:16] data and that time series data streaming in with this more informational data we can we we want to get that data up into connect and so in the middle box there you have an agent that I briefly mentioned previously running on the
[09:29] Windows platform we install that and then we can go into connect and configure what objects or what data do we want to bring in. So we obviously want to bring in the hierarchy and bring in all the data for the trucks, the metadata and the time series data.
[09:43] And so, that middle step we decide um how what how far back we want to go and we can just configure that and then data starts flowing. What's important about that middle step is that that's managing uh the the updated flow of data for us.
[09:57] And then that would then we get that data into connect. So, now we can jump into the connect platform, uh Aveva SAS platform, and have a look at the data and verify it's transferred data before you uh we utilize that. So, we can do a
[10:09] query. We can look at our truck fleet. This is a a summary asset giving us a high-level view of what information that uh that we have. And it's just giving us some titles around all's uh for all all
[10:21] the trucks that we have available. We can drill into a specific truck, look at some information about that, check to see the what relationships it has, and then we can view some data uh some
[10:33] metadata for that truck, for the time series, just validating that we've got the data in this platform. And this is where users might use this data as well for uh other use cases of or maybe sharing data with uh other people as well. Next, we want to create a data
[10:46] view, which is a data shape we want to make available for our users in Databricks. So, with that we can set up a shape, we give it a name, we can set specify one or more queries that we want
[10:59] to define what that data shape what is that table going to actually look like. So, we can search for the uh the equipment. In this case, we're dealing with trucks. Uh so, we're going to do a search for our trucks. And the but this
[11:11] search is dynamic. So, if we add more trucks later, they'll automatically be added. We don't need to come back and rechange this data shape every single time. So, once we we we define what we want to use in our data shape, then we
[11:24] can go and customize what fields or what columns would we expect to see for that. I think there's about 249 here. So, we'll filter those down and focus primarily on the time series data. We
[11:37] can then also add summary information as well. So, if we don't just want the raw data coming in from sensors, we could add in add in counts and totals, and also summaries. Also, there's a weighted mean there as well, in case you need to do like a rate calculation down down the
[11:51] track, you can do that. Now, we get our view up. We can just make sure it's a representation that that we're happy with. We can do some custom customizations here. We can change within interpolated to stored, in case
[12:03] you want to see the actual raw data for that. We can then specify some timing information for the new default queries and what what data shape and time we like to see for that. And then there's the
[12:15] usual suspects of being able to update the column names or customize the naming of them or reorder those as we need. So, you can create one or more of these for a variety of
[12:29] uh use cases. So, once we've got our data view, the next step is, well, we need to set up a catalog and then also add a table so we can share that data with Databricks. And setting up the
[12:41] the catalog or the data sharing is very straightforward, kind of like three steps. So, on the top right, we go into our Databricks environment, we get our sharing identifier, we then take that to the to the left-hand side to connect,
[12:54] where we create a new share, and then we put in the name of that. that. And on the right-hand side, in step C, we go back to Databricks, that share that appears, we can associate
[13:06] that share with a Databricks catalog, either a new catalog or an exi- existing catalog. So, what that means then now we we we've got that share connection set set up. So, now, on the connect side, we
[13:18] can create tables, and those those tables will appear on the Databricks side. And again, the point here is the tables will be in Connect, we'll be updating those. If from Databricks, I will send it out as black and just query the data. I
[13:30] don't have to copy the data to get access to that. So, with that, we want to create that table that is going to be available in Data Bricks but is in Connect. So, we specify
[13:42] that data view that we created before. And we give that it obviously a name. We do some configuration. We decide whether we want interpolated value or stored values. Interpolated is very common cuz a lot of applications like to see a
[13:55] value at a specific interval cuz it's easy to process. But the If you want, you can also use stored values. But the challenge for some applications with stored time series data tends to have different alignment. So, you won't
[14:07] always have a value at every time stamp. So, it'll be kind of sparse data set. So, it makes sense to do interpolated. Then we specify the time range. We could do a fixed time range. Maybe 2 years ago, we've got some events that occurred. We want to do some modeling
[14:20] based on that time set and that time range. So, we could do a fixed time set and make that data available as a table. But in this case, we're just going to do an extending table whereby we just say a start time when we want the table to first have data. Now, we'll just keep
[14:33] adding to that table as new data arrives for that. So, now we've got that table created. We now go and associate that with
[14:46] our line of our shares. And in this case, this is the share that we created with our Data Bricks and our Delta available. Then we've got a summary of information there. We get the interpolated values every 15 minutes and then every hour, we're going to do a
[14:59] refresh and update data update that table. So, for all that streaming data we've got from all those trucks, we'll get new values every 15 minutes for that. So, once we've got the virtual table, we can now jump into Data Bricks. And so,
[15:12] in Data Bricks now, we see that table that we first we set up. It's part of the Aviva Lighthouse catalog that share that we set up. We can see that we've got our fields. Check that
[15:25] any information looks correct. It's exactly a representation of the table we corrected. And now we can look at some of the data from that. And again, importantly, the data hasn't been copied to Databricks. Uh uh the data is being
[15:37] queried directly from uh over the share from uh from from Connect. Then the next step, of course, is to bring that operations data with that LIMS chemical analysis and do that simulation. We're not
[15:49] showing that cuz that was done in the customer environment. So that was step zero. The output of that, of course, is the temperature. And that temperature we we want to get back to Connect so we can install a library. Uh uh AVEVA provides a Connect library for Python. So we can
[16:02] just install that Python library. Then we can specify some configuration information for that library. Then we can go through the process of making a connection to uh Connect. And then uh
[16:14] the next step, of course, is writing some values. So the first thing we'll do is we'll specify a type. The type is just a shape of data we want. So it'll be a timestamp and value for this cuz we want to store temperature. And then
[16:26] we'll create a stream. And the stream will be that actual temperature value that we're going to write to that stream. It'll be a streaming data every time we write updates for that. And then just an example here of a cell where we're showing how it's very easy using
[16:38] the Python library to be able to insert values and just uh retrieve those values. So every time they go through this process of taking that operations data with the LIMS data,
[16:50] running that simulation model, the output that temperature, they can then write back to the AVEVA platform. So once the cus- customer's happy with this notebook, they can then schedule that. So that's what the customer did. They scheduled the notebook round about every
[17:03] 2 hours where they'd run the simulation and then write the data back into the Connect platform. So if we go back to our architecture, the last step, the data is now flowing through. Uh the data is back in Connect.
[17:16] It can obviously be used in Connect for visualization, for other analyses, and all use cases. And the last step is that lower PI agent, which we configure from Connect to write those
[17:28] values, those temperature values, back into the Aveva PI system for the operator. That's step number eight. There's an example visualization on the right of how an operations user, they're used to being in environment with PI
[17:40] Vision. They're not going to log into Databricks. So, it makes sense for them to have those values show up for them in their PI Vision screen. Then they can make options and decide if they want to make changes to temperature to see if they can improve that
[17:54] uh that ni- that nitrate production. So, the wrap-up of a lighthouse, we always do do kind of like a summary, what did we achieve? So, again, our goal was to improve that improve that nitrate
[18:06] performance. We were able to bring products together for the solution, and they had some great results with that. They were able to increase their their nitrate yield. There was also some additional as well. You can
[18:18] see this at the end around efficiency and otherwise. And then the quote from Matthias, the end user, the point that he's now got this platform set up, and he can use that platform going forward. They've already implemented, I think, about eight or maybe almost 10 other
[18:32] data views. So, there's now about 10 tables now they're accessing via data sharing. They're starting to add more uh scenarios and use cases for that. I'll just jump to this another slide in
[18:45] a sec. So, just to wrap it up, you know, why would you do this? You might think, "Well, I could do this myself. I could code this." And certainly you could, but we believe that with Connect we make it a lot more simpler. We make it easy for an operations user to get data from a
[18:57] number of Aveva products or third-party products into Connect. They can apply one or more use cases in Connect. And additionally, they can do data sharing with users in Databricks. We had working
[19:10] with another lighthouse customer, they were very advanced development shop on Databricks, but they preferred the solution on the right cuz they could let their operations teams decide what day they wanted to move to connect. They
[19:22] could work in connect with these cases, and then if they wanted help from the developers in Databricks, they could then make data available by the share, and then the uh these development teams could then
[19:34] just help and take the data from there. So, it simplified their work workload as well. Whereas on the left-hand side, you got to deal with copying copying data. You got to deal with out of order order data and other issues. Well, we just make that stream. We deal with all those
[19:47] issues for you. We just make make the data flow seamlessly. I made this slide on Delta Sharing in case you're not familiar with Delta Sharing specifically. Uh the idea is again that zero copy. The data is
[19:59] available in connect and we're just querying it from Databricks. You don't have to make copies of the data. It's always up-to-date. It's it's a read-only copy as well, and just simplifies uh end users having to use that data.
[20:14] And last but not least, uh we have a booth uh 5653. It's right near the puppies, so you can see puppies and talk about Alteryx Connect. What What What else would you rather be doing today? So, please stop
[20:26] by the booth if you'd like more more information. You know, and basically the story is that connect, we understand the industrial landscape. We understand the industrial use cases. We can also enable users in data with uh as well for those
[20:38] solutions. So, uh about 10 seconds left. Thank you very much for your time. I'll be around if if if you have questions. We'll also be available at the booth later today for the rest of conference. So, enjoy the rest of your conference. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.