Sponsored by: Datafold | From SAP HANA & Talend to Databricks in Under a Year

Summary

Evri, the UK's largest parcel delivery company delivering over a billion parcels per year, faced a migration of 1,200+ legacy objects across SAP HANA, Talend, and five fragmented BI tools with hard contractual decommission deadlines that made a 2–3-year traditional rewrite impossible.
Datafold's AI Migration Agent automated code translation to PySpark and Spark SQL and used a data knowledge graph to map all dependencies across systems, reducing 2 million lines of code to approximately 100,000 in under a year with zero disruption to operations.
Automated data diffing validated output equivalence at the value level across thousands of datasets side by side over time, replacing manual verification and ensuring migrated pipelines matched legacy system results before decommissioning.

Chapters

FAQs

Who is Evri and what made their migration to Databricks so challenging?

Evri is the UK's largest dedicated parcel delivery company, delivering over a billion parcels per year with approximately 30,000 couriers and 12,000-plus parcel shops and lockers. The migration challenge included 1,200+ legacy objects across SAP HANA, Talend, and Power BI, undocumented code, lost institutional knowledge, near-real-time operations with sub-10-minute SLAs, and hard contractual deadlines on both legacy platforms.

How did Datafold's AI Migration Agent accelerate the migration?

Datafold used a data knowledge graph to map all dependencies across systems—capturing undocumented code relationships—and then automated code translation from the legacy stack to PySpark and Spark SQL. The AI-assisted approach reduced 2 million lines of code to approximately 100,000 and completed the migration in under a year at a fixed price with a guaranteed timeline.

What is automated data diffing and why was it critical for Evri's migration?

Automated data diffing performs side-by-side output validation of the migrated and legacy pipelines over time, comparing results at the value level across thousands of datasets. This replaced manual verification and gave the team confidence that the Databricks pipelines were producing equivalent results before the legacy platforms were decommissioned.

Why did Evri choose Databricks over Snowflake for their platform migration?

This video covers a platform selection process in which Databricks won out over Snowflake for Evri's migration. The specific selection criteria are covered in the video but are not detailed in the transcript excerpt; Evri's data-driven operations—including near-real-time parcel tracking, courier optimization, and geolocation analysis—informed the evaluation.

Full transcript

[00:08] Hi everyone, welcome to a session about migration. If you made it here, you probably care about this topic quite a bit and uh we're excited to share the story of how every migrated to data bricks and how we enabled this from a
[00:20] technology standpoint. My name is Gleb. I am CEO and co-founder of Datafold. We automate migrations to data bricks. Our booth is right there if you're interested. And um I have a pleasure to share the stage today with
[00:32] Har Adwal who is chief data officer at every. >> Hi everyone. I'm Harindra and Clebs has mentioned I'm the chief data officer at
[00:45] every I'm guessing a quite a few of you may not know who every are but we're actually the UK's largest dedicated parcel delivery company. Uh we work with uh a lot of major brands. everyone from
[00:57] Amazon to vintage. Um we're actually a group. So um there's every which is the core brand. Um and that's the largest part of our business. Um but we recently merged with uh DHL's
[01:11] UK e-commerce uh business. Um we acquired Colate which is an Irish logistics and customs clearance specialist. Um and as part of the the the merger with DHL's UK
[01:26] e-commerce business, we also integrated with UK Mail um who are a leading uh business letter delivery company. Um so
[01:39] we um essentially want to make it as convenient as possible for people to send and receive parcels. So we have uh an extremely extensive network throughout the UK. So we have uh 30,000
[01:52] um couriers who deliver our parcels. Uh we have 15,000 um plus or sorry 12,000 plus parcel shops and lockers spread out across the country. Um and they're all serviced by
[02:06] a state-of-the-art network of hubs and depots. Now um we deliver over a billion parcels and a billion letters per year. So as you can imagine um delivering all those
[02:19] parcels and letters generates a huge amount of data uh which we use uh to make better decisions. So we're a very datadriven company. Uh we use data extensively to optimize the network,
[02:33] ensure that our parcels get delivered on time and to make commercial decisions. Um so just some examples of uh how we uh use that data. So um uh operations team
[02:46] uh ensure that we have industry-leading um speed of service delivery and they use the data pretty much 247 to make sure that parcels are on track to be
[02:58] delivered within SLAs's. Uh we do a lot of work uh within the final mile to optimize that. Um so typically we'll look at things like um geoloccation analysis to make sure our parcel shops
[03:10] and lockers are in the right places. that we're paying couriers a competitive rate for the local market. Um, we do a lot of work around commercials to make sure that we're predicting which clients
[03:23] may be at risk of leaving so we can proactively retain them. Um, we do a lot of work around um what we call rounds, which are territories for couriers and drivers. Um, transport optimization to
[03:37] make sure our vehicles are as full as possible. Um and that's obviously good in terms of cost but also the environment. So we're generating fewer CO2 emissions per parcel.
[03:49] Now um what we realized was that um the business was growing rapidly. We had a lot of ambition. Um we had a lot of data use cases but our technology stack as it was uh 3 years ago was just not capable
[04:04] of supporting what we wanted to do in the future. So like a lot of companies uh we uh have kind of built up a quite a complicated ecosystem of uh data
[04:16] platforms and tools and each on their own in isolation made a lot of sense right so kind of we had a data Oracle data warehouse uh for our batch commercial reporting um and financial
[04:30] reporting um the operations team brought in SAP Hannah u which is a very fast in-memory database for near realtime analytics so they could keep an eye on performance in trade. Uh we had lots of
[04:44] different BI tools coupled with those data warehouses. So we had business objects coupled with Oracle, we had SAP analytics cloud, uh we had Tableau, we had PowerBI. So um and then we we had
[04:57] talent as well which is like an ETL tool um for data engineering. So we had this uh kind of ecosystem or a hairball as I I like to call it um which really was
[05:10] starting to constrain us and also didn't have some of the capability that we needed. So we had increasingly more sophisticated use cases are required uh data science and operational research. Um and what
[05:23] our data science teams were having to do was extract data, create models locally. Um but then we had no easy environments to productionize it. We had data um duplicated across all
[05:35] these different platforms. We don't really have large data engineering teams. So they were having to do a lot of firefighting across many many different platforms all at once. Um these a lot of these tools kind of predated GDPR. So the governance tooling
[05:49] wasn't where we'd like it. Um and so all in all um we needed to to move on. So we looked at a market. We went through a whole procurement RFP process.
[06:01] Uh we kind of whittleled it down to Snowflake uh versus data bricks. Um and we did kind of like pilots with each. Um and data bricks really ticked all the boxes, right? It could handle multiple workloads, data lake, data warehousing,
[06:15] data engineering, data science. It had unity catalog. So it gave us better better governance. Um it integrated really well with some of the future tooling that we had like um uh PowerBI
[06:27] which is what we want to standardize our BI around. Um so we decided to to make the move uh and centralize everything onto the data bricks platform. Um but obviously as you as you know
[06:42] migrations are always a huge challenge. Um they're never easier than quite a few um in my career. Um, and all they do is generate um, a lot of sleepless nights.
[06:54] Um, and there was a lot of extra pressure because we had upcoming renewals uh, for some of these legacy data warehouses. So, we had multi-year contracts. We knew not only was it going to be expensive uh, to renew um, but
[07:07] also we would have like dual running costs and we really wanted to to avoid that um, we absolutely wanted to avoid any sort of extension. Um we had the issue where many many
[07:19] people had worked on these uh platforms and data warehouses over time uh but have since left the company and they taken that knowledge with them and in most cases uh the code was the only documentation that we had. Um and then
[07:32] within those environments because they had been built up over the years there was a lot of complexity. Um so even you know one of the lineage graphs we tried to create for a report running off SAP Hannah took I can't remember how many
[07:45] weeks to produce it was just literally store procedures layered on top of views layered on top of store procedures and so on. So it's extremely complex. So this was uh a really really big
[07:57] challenge. Now there's obviously different ways you can approach migrations. There's the burn it with fire approach where you go I'm just going to start again from scratch on the new platform. Um but the
[08:10] challenge you have there is like you know you have to spend a lot of time analyzing what you previously had modeling um and recoding everything on the new platform and you end up with two
[08:22] different outputs and then you have to then kind of figure out well was it the old logic that was wrong on the old platform or have you introduced new bugs on the new platform because the difference between data engineering and
[08:35] software engineering is you always have to worry about changing data as well as changing code. Um you can do a strangler pattern so steadily migrate functionality across but we didn't have
[08:47] the the luxury of time to do that. Um, so we took the other route which is we'll lift and then optimize later, which on the face of it sounds like the worst option, but actually typically it
[08:59] turns out to be the best one because at least it gives you a chance to be off the old platform and you don't end up with a whole load of orphan data warehouses like many companies have and abandoned migration projects.
[09:11] Um, but it still left us with huge challenges. um and my head of data engineering at the time Chris Hopinson um he said he'd come across a company called dataf fold um and he had glare bond I think it was a data engineering
[09:23] podcast and they had a pretty novel approach using AI coupled with their data diff tool um so we had those initial conversations um and it sounded promising but we
[09:35] didn't um we obviously wanted more than um uh obviously um you know uh reassurance verbal reassurance so we did a pilot Um and that pilot was promising. Um and we thought this is probably a the
[09:48] best way and the most modern way to approach migrations uh which is let's use AI uh coupled with automation technology coupled with data automated data diffing. Um and the benefit was
[10:01] that they were offering us uh uh guarantees around price and timelines. Um so all in all it was a promising solution and that's the path we went down. So I'm going to hand over now to Gleb who's going to explain uh the exact
[10:14] process. >> Thank you Harender and uh thank you for your trust in uh trusting us with a very complex migration. So when it comes to actually executing migrations,
[10:27] if you have done one or you've seen your colleagues in the industry done one, you know that it's a usually a very complex process that almost always goes over budget and over time. And as Harvard mentioned, one of the most important
[10:40] decision criteria for every to choose dataf fold was that we work always on a fixed price and guaranteed timeline outcomebased delivery. and promising that and taking that risk is actually
[10:52] quite challenging. And so I think what's interesting to talk about is what kind of technological innovation actually enables us to take that risk and actually deliver on an outcome based uh
[11:04] basis. So one of the biggest challenges with migrations is that they span multiple systems and they involve a lot of objects as Har mentioned a lot of them are probably undocumented or
[11:16] documentation may be out ofd and sometimes even the business and data users don't even know what's in there and what's the purpose of certain types of jobs and so that's why for proper AI
[11:28] automation it's not just enough to have an AI agent that plows through all this code that probably will end up with a lot of missed requirements and not a great data quality and not a great
[11:40] efficiency. So the really the first step is building what we call a data knowledge graph. Data knowledge graph is very simple in context. Basically the idea is that is we parse all the codebase and all available metadata
[11:53] documentation and context for the legacy platform. So we can understand and map out how the data flows end to end. So lineage right we understand how every piece of source code every script
[12:06] whether it's shell or SQL or in the case of talent XML how it all interconnects with the actual data assets that are produced are there any important considerations about how the data is
[12:18] used for example in every case um a lot of the business use of data was centered around Tableau PowerBI dashboards and so it was really important that the migration accounts for end to end
[12:31] delivery of those assets which is again important to track across um multiple platforms and the data knowledge graph is built by an AI agent automatically by parsing through all the source code and
[12:43] once we have that we can actually go to the next stage which is actual translation and the actual translation conceptually is quite a simple process. So once we have the semantic knowledge graph, we can then take every bit of
[12:57] legacy code and for every asset that we want to move over to data bricks, we translate the code using LMS to data bricks, whether that's pispark or spark SQL. And that is only I would say 20% of
[13:12] the challenge. The bigger challenge is how do you ensure that once you have the translated code, it produces the same output and consistently produce the same output as the original job. Because
[13:24] otherwise, unless you do that and unless you prove to the business that the output of the data is the same, it's pretty much impossible to get user acceptance. And without that, it's impossible to execute the migration and
[13:37] complete the migration. And so once we have the new code, the agent automatically runs it on the new platform. And then it does what's called a data diff. And the data diff is essentially a comparison between the
[13:49] legacy platform and the target platform on the value level across every single data set. And that um provides us with a feedback for the agent in terms of okay are we actually reconciled? Are we
[14:01] producing the same output or not? And then in the context of migration, data diffs are used within the agent as it executes the code and tests it. But we also use them over time as monitors because we want to run the system side
[14:13] by side over let's say two or three months to make sure that over time with all the incremental processing and complex logic the systems don't diverge before doing the final cutoff. And data diff again is consumed by the agent
[14:26] programmatically but it's also exposed to the business users as a visual output that they can see. And in a real migration, obviously the great outcome is that everything matches perfectly.
[14:38] But in a real world, many assets won't match perfectly because you have different source data or different handling of data types. And what's really important for the success of the migration is being able to
[14:52] explain those differences. So tell the business users that this is what's different. Everything else is the same and this is why the difference, this is where it comes from. Maybe it's rounding error. Maybe it's difference in the source data and so that allows us to
[15:05] have the loop where every data asset is transformed automatically and the agent can work on this autonomously because it has the context of the knowledge graph. That's why we're able to spin hundreds or thousands of agents that each work on
[15:18] a subset of the migration in parallel and only escalate errors to the human where the human input is necessary versus having the human to drive every single bit of the process.
[15:30] And one other important consideration for a migration to a platform like datab bricks is that it's not enough to just make it run right so we can transpile code from let's say SAP HANA SQL to data
[15:44] bricks SQL but will it run efficiently will it provide good enough total cost of ownership will it meet the SLAs of the business the
[15:56] expectations in terms of data timeliness they hard questions because for systems like talent and SAP HANA their architectures and the patterns that are used there in terms of store procedures
[16:09] incremental processing microbatching they don't map to data bricks one to one and if we try to execute a store procedure that loops over 10,000 rows at a time on data bricks the performance would be terrible so not only we have to
[16:23] make it runnable on data bricks we also have to use the right patterns for each legacy data object the Good thing is that data bricks offers really great primitives such as lakeflow jobs SQL and
[16:37] lakeflow spark declarative pipelines which is one of the newer additions that essentially allow you to create end toend pipelines in pure SQL declaratively and is a really good way to refactor a lot of the legacy store procedure code that processes especially
[16:50] event data and the interesting thing is that we started the migration with close to 2 million lines of legacy code and in the end we ended up with about 100,000 lines of code on data bricks. So what
[17:02] that means is that if we properly modernize the system that you end up with on data bricks is actually much much simpler. It's fewer code. It's easier to maintain and it runs pretty
[17:15] much as good as uh as legacy with some tuning. Um, so that's what essentially AI fully automated migrations enable you is to migrate all your legacy with fixed price
[17:30] guaranteed timeline and not just make it runnable but actually properly modernize it onto a platform like data bricks. All right, I think we are up for questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.