Hays and Databricks Build an AI Candidate-Job Matching Engine

Summary

Hays and the Databricks FDE team built a single AI matching platform with two tuning profiles — precision-optimized for specialized white-collar roles and speed-optimized for high-volume blue-collar recruitment — reducing candidate search latency from 20 seconds to below 3 seconds.
The architecture uses an ESCO-anchored ontology for multilingual skill normalization, deterministic extraction, embeddings, BM25 keyword search, and reciprocal rank fusion for candidate ranking, with human feedback continuously incorporated to improve match quality at global scale.
Unity Catalog provides token-level data lineage and MLflow supports model versioning and drift detection to meet EU AI Act requirements, bias evaluation standards, and a roadmap targeting sub-500-millisecond performance for global deployment.

Hays and Databricks Build an AI Candidate-Job Matching Engine

Hays and Databricks built an AI-powered candidate-job matching engine that searches multilingual CVs at global scale. The platform balances precision for specialized white-collar roles with speed for high-volume blue-collar recruitment, reducing search latency from 20 seconds to below 3 seconds while targeting sub-500-millisecond performance.
Learn how the Databricks FDE team and Hays use an ESCO-anchored ontology, deterministic extraction, embeddings, keyword search and reciprocal rank fusion to normalize skills and rank candidates. The architecture incorporates human feedback, bias evaluation and EU AI Act considerations, while Databricks Unity Catalog provides token-level lineage and MLflow supports model versioning for governed, explainable recruitment AI.
Hays AI candidate matching session: https://www.databricks.com/dataaisummit/session/ai-candidate-matching-lakehouse-transforming-recruitment-operations
Databricks AI Search documentation: https://docs.databricks.com/gcp/en/ai-search/ai-search

Chapters

FAQs

What are the two recruitment matching problems Hays needed to solve on a single platform?

White-collar recruitment requires precision because skills are specialized and the right candidates are rare, demanding a high-accuracy ranking system. Blue-collar recruitment requires speed because lead times are short and competition is high, demanding a low-latency, high-throughput retrieval pipeline. Hays built both as distinct tuning profiles on a single Databricks-based matching platform.

What is the ESCO taxonomy and why is it important for multilingual CV matching?

ESCO is the European Skills, Competencies, Qualifications and Occupations taxonomy, which provides a standardized multilingual vocabulary for skills and job roles. Hays uses an ESCO-anchored ontology to normalize skills extracted from CVs written in different languages, enabling consistent candidate ranking across global markets without language-specific pipelines.

How does the Hays matching engine handle explainability and regulatory compliance?

The system uses token-level data lineage through Unity Catalog so that every matching decision can be traced back to its underlying data sources. MLflow supports model versioning and drift detection, and the team incorporates human feedback and bias evaluation as part of their compliance approach to the EU AI Act.

What latency improvements did Hays achieve with the Databricks matching engine?

The matching engine reduced search latency from 20 seconds down to below 3 seconds, with a roadmap target of sub-500-millisecond performance for global deployment. Reciprocal rank fusion combines keyword and embedding retrieval signals to maintain accuracy while meeting the speed requirements for high-volume blue-collar recruitment.

Full transcript

[00:07] Um, yeah, hi everyone. My name is Sebastian. I'm with Hays PLC. We are globally active recruitment company. And obviously, we are interested in bringing the right people to the right jobs. And to do that efficiently, we need to rely on a very efficient search and matching match engine, which we can
[00:24] deploy at a global scale. So, in order to reach that, we teamed up with the Data Bricks FD FDE team. And yeah. We help the We help now to put thousands of people into jobs every working day
[00:40] and match candidates to the roles which they need. All right. And over to Chase. Thank you very much. And just a quick intro from myself. Chase Edwards, I run the AI forward deployed engineering team in Data Bricks for EMEA. So, our our team focuses on the productionization of
[00:57] AI systems at scale. All right. So, maybe to characterize what we are facing a bit more definitely, we are basically solving two matching problems on one platform.
[01:13] White-collar recruitment needs precision because skills are specialized and the right candidates are rare. So, think of a classical example for white-collar recruitment case of data engineer, for example. Um, blue-collar recruitment on the other hand needs speed because lead times are
[01:28] very short and the competition is high. So, we're talking about better substit- of more substitutive profiles. So, this one we built here is a retrieval engine with two two tuning profiles. One is precision optimized and the other one
[01:45] is speed optimized. So, core challenges are finding the best available candidate in a high-dimensional optimization problem. It's the white-collar one, more analytically complex. And the other one is fill the role before the competition does. So, there we are facing a strong
[02:01] speed and automation challenge in order to protect our margins. There we go. All right. And basically, as I already said, matching is the business with us. So, candidate to to
[02:16] job matching is the core engine of a global recruitment companies company. So, basically, this is then driving three levers which are directly impacting our P&L. One is recruiter productivity. So, minutes spent searching are
[02:31] minutes not spent placing people. The second one I already elaborated on. We want to achieve a reasonably high level of fill rate accuracy. So, the better the short list, the better our chances are to win the placement. And the next would be time to match. So,
[02:47] speed speed wins the candidate in both scenarios while being more pronounced in the blue collar scenario. So, here we are looking at, say, acceptable lead times below 24 hours in some cases in a high competition market.
[03:07] Yeah, and basically, um the starting point was a 20-second search across messy multilingual CVs. And the North Star for us is to bring latency below 500 milliseconds. Um on a governed explainable
[03:22] uh um AI and information presentation set which our recruiters must trust. For us, what's also very important is we are people business, meaning trust matters a lot. It actually matters as much as speed. So, if recruiters working
[03:38] with us do not understand the results our search is providing them with, they will not adapt and they won't will not use this solution. So, we need to build in explainability not only for uh, uh, reasons, but really also to win the trust of our own recruiters into this
[03:55] product. Yeah, and basically, um, we have also a set of four core principles before we basically get to
[04:13] work on that. So, one is anti-bias bias and compliance by design. Obviously, labor markets are uh, strongly relying on bias-free decision-making, which is also a core mission of our product. So, we as Hays believe in bringing the right
[04:28] people to the right job irrespective of any any bias, uh, uh, um, concerns or whatnot. Um, second I already ex- uh, elaborated on. Consultants must trust and we can only achieve that trust via full
[04:43] transparency. So, we cannot have a scenario in the sense of, and now I'm doing a little Britain uh, pun, "Confucius says, yes is not an option for decision-making in our company." Um, third would be search efficiency at at scale. We, um, are globally active and we
[05:01] digest about 15,000 CVs per day in the pipeline entry. Meaning we have to achieve a high level of uh, efficiency in order for our consultants to be able to tap into the entire knowledge base in order to find and surface the right candidates and
[05:17] also the right jobs. And second, fourth would be human in the loop, not human on the side. So, our consultants actively evaluate search results because in the people business scenario, they are the ones who are propagating basically the results of this product. And they flag
[05:34] drift and tune weights, so the model improves continuously by them using it. So, it's classical reinforcement learning uh, scenario actually. So, build it right from the first table and the recruiter and the recruiter will trust what comes out of it.
[05:50] Uh it's important to think about this in the context of FDE thinking about building products for users. We want to design with the user in mind as we establish and build these products, take them along the journey, make sure given that they're a human in the loop that they're designed for the CUJs they're
[06:06] designed for. We have cohorts defined that are going to evaluate the alpha and beta and V1 products that we that we ship. Um what we wanted to do is start simple. We had an iterative approach to the
[06:22] building of this system. And so we began from a early stage starting point that the Hays engineering team and product did a lot of great work on and wanted to expand upon that into a fully fleshed out product. Uh so we wanted to start with a a simple
[06:37] path to production on a limited subset at first and then scale from there. Core engineering principles, fundamental principles there. Uh we wanted to explore expose scoring decisions. So as was mentioned by Seb, there is a fantastic amount of
[06:53] regulation and compliance across a variety of different regional domains and things like the EU AI Act which we had to spend a lot of time evaluating and understanding the technical implications of uh in in the recruiting space and make it a tunable parameter as
[07:09] well so that it's something that isn't static for any of the recruiters that are using the tool. And then lastly, evaluating early and often. We spend a substantial amount of time building a robust eval framework and the evals themselves alongside the the recruiting analysts and
[07:25] organization. In that way we can build something that has a degree of confidence early stage and then we can iteratively improve across what would be many different technical decisions as the product is designed.
[07:43] Yeah, and that basically leads us to the first um challenge here, spinning up the right taxonomy for it. So, concepts like taxonomies and ontologies are interchangeably being used also at this conference here, I learned earlier today. So, um for us really, think of it as
[08:00] follows. So, boiling it down to a very simple example is somebody lands a CV with you where and they might be doing that because they are data engineer and a quirky person, they do not list Python as a programming language but list Python 3,
[08:15] Pi, Pi Python programming or whatnot. So, what we need to do in the first step is uh and and and we are tackling that basically with a four-stage resolver that normalizes multilingual CVs and puts them into an ESCO anchored concept.
[08:32] Who of you knows what ESCO is? So, it's a European standard ontology for job titles, so to speak. So, it's being used by the International Labor Organization and we use that as an anchoring ontology in order to seed our own ontology, if you want to say so.
[08:50] And then, basically, we extract aliases, so direct string matches against taste alias tables, zero LLM cost. Then we do an ESCO lookup and we map basically to 3,000 plus applications and 13,000 skills across 28 languages and
[09:08] then use fuzzy and embedding in order to be able to um say say um either find a direct say keyword-based match or really do an embedding in order to um assign to that um seed vector space, so to speak, using
[09:23] the semantic load of an item we extract. And then, basically, we have an unknown queue, which is basically un- resolved terms. So, that happens because there might be a new programming languages language coming up, which is neither contained in the body of an LLM or in
[09:38] that ontology already, meaning we do not have silent drops. So, a human will always review that unknown um queue and then assign it accordingly. And this then leaves us basically um with a standardized um um token which
[09:54] you see in the lower line here, which would be skill 0042 Python in that case. So, we generalize everything into that based on either semantic load or keyword matching. It's also difficult to overstate just how much variety that you might see in a
[10:11] CV. All of you, I imagine, have seen fantastic CVs, maybe not so great CVs at times, and the kind of differences that you might expect not only in the context of, let's say, a single role, but also region, language. It expands very, very quickly.
[10:33] Yeah, and this is my favorite slide because it basically by showing you how overcrowded this slide is, this basically shows you how say, complex the challenge is. We have at a at first sight very easy and simple first step because actually we are heavily doing the work in the loading
[10:49] stuff and normalizing it before we do the search. So, actually the magic happens quite left in our pipeline. So, first um we derive an Esco normalized concept, as I already said. So, we have one CV and we have multiple type chunks we extract from that CV. So, skills,
[11:05] experience, qualification, and an all chunk which we then embed separately with a chunk type label. So, this basically is what we take out there. Then we have the type chunks, and then we we within those type chunk spaces we collapse basically uh synonyms in that
[11:22] vector space within those type chunk vector spaces. Um what I also just elaborated on. Um and what you see here, and this is the uh second lowest line here, um, we achieve already a 92% skill
[11:37] normalization rate in the extraction pipeline, which is very important for us because we remove a lot of unfavorable noise basically from our knowledge representation, which makes say the later stages of the search engine, so the actual search algorithm, much easier to build because we did a lot of the
[11:54] work people might be doing in the search algorithm itself itself. So, we do the hard work early on. Um, and this led us actually to a lower than 3 seconds latency already in this very early stage of the product. So, the alpha testing version of that product has been shipped
[12:10] in Australia and New Zealand just Monday. Um, so we built that within say 3 to 7 months together with Databricks FDEs. Um, and we are already at a latency below 3 seconds searching on the full set of CVs right now. Um, and basically
[12:26] what we then have is we have two signals fused, but I'll let Chase elaborate on that. Absolutely. Super excited about getting that into prod. Uh, I'll talk a little bit about hybrid retrieval. Um, the idea here is we wanted to take a
[12:43] number of different uh, chunk type labels across skills, experience, qualification, and then all. Uh, and we wanted to be able to understand the data that lies underneath that and do semantic search uh, as well as uh, uh,
[13:00] uh, keyword search on top of that. So, we use the labels for the keyword searching plus the um, the uh, uh, semantic search side of things. Uh, and so, what we did is we combined those two things with reciprocal rank
[13:16] fusion. Uh, we grouped by candidate uh, the best score per chunk type as well. Uh, and then we summarize that together across both a normalized weight and a best score. And the idea there is you know there within the context of
[13:32] recruitment data there's a substantial amount of exact identifiers, certification names, job codes. So keyword precision really does matter. It adds a lot to the search. However, we still want to retain some of that semantic similarity, right? That that
[13:49] vector stores are so good at. And so we took the best of both worlds and combined those together. And then what we do and you can see that at the bottom there is we allow for a parameterized waiting across the different types. So you can see we have
[14:04] 50% under skills, 30% under experience, and 20% under qualification. We allow the users to make their own determination around what that hybridization should be across different waiting. So if you want a little bit more weight depending on what type of
[14:19] role you're trying to fill and experience matters a whole lot more on a on a role like that, then you can weigh it appropriately. Now working through an example of what this looks like in practice,
[14:35] uh walking through how we want to reward matching. You know, what is the outcome that we want to achieve? And so for here at starting the upper left, we want a coverage multiplier. We want to reward a broad set of potential
[14:52] matches that can come back to a recruiter. And the idea here and you can see we've got a 1.0, 1.1, and a 1.2 waiting across a single type match, two type match, or three type match.
[15:07] And so the idea here is that we don't want to penalize the ability for the system or the product to return many different matches and then gives the recruiter the opportunity to evaluate those. On top of that we also wanted to
[15:23] normalize across missing data. Uh, often these CVs don't have all data included, and so we wanted to find a way to normalize this. Candidates in the original version of the system were penalized unfairly for having data absent, so we corrected for that uh,
[15:38] through weight renormalization. Um, and then lastly, uh, we wanted to make sure that, and you can see across four different example candidates there, that we did that across multiple different match types.
[15:53] And so you can see on the right the normalized for missing before we included it and then after uh, the candidate C that didn't have the additional data included has now been ranked differently as a result. And I think the importance around fairness here is is super critical,
[16:10] right? On a regional basis from not only the mission statement that that, uh, Seb highlighted earlier around how Hays does matching and recruiting, but also, uh, in the context of compliance for things like the EU AI Act.
[16:29] And then the last bit, uh, around there were many different technical decisions and many different ways you could design and build a product like this. And originally, we had considered a decomposition method where instead of doing a single search, we split it into different searches by label type, which
[16:45] I mentioned earlier, those four different ones, uh, the metadata labels. Um, there were a number of challenges and tradeoffs to consider there, primarily around cost and latency. Uh, low latency is super key for a product like this. You imagine the number of times that this needs to be run and how
[17:01] many users are using it. You want this to be a rapid experience for them. On top of that, we want it to be budget-friendly. There's a lot of conversation, uh, today about cost in the LLM space, and we don't want to be making, uh, four different LLM calls plus a supervisor call that routes. And
[17:18] so, we quickly uh, iterated and shifted away from that methodology and lined up on a single single design for retrieval.
[17:37] Yeah, and maybe getting back to trust being the product. Um as I already said, the engineers only have an incomplete idea about what constitutes basically a suggestion being made by a recruiter because there's a lot of implicit knowledge in
[17:52] the heads of our recruiters at at this point. So, basically, we have to ensure for three core components. So, first question would be is the knowledge base correct? Meaning, we do
[18:08] a very exact and granular matching in and sorry, matching in the very exact and granular extraction pipeline which we can also track back to the token level. On the one hand, this is important for the trust component, but I will elaborate on that later. This is
[18:23] also important important in many regulated labor markets. So, for example, in my home country Germany, you would not be allowed to use this technology if you could not trace back in a placement decision until the first step of data extraction actually.
[18:39] So, second part, are the right candidates returned? So, implicit knowledge also opens always the room for say cognitive dissonance. So, it might be that very experienced recruiter might surface people or profiles which are super different from what they would be
[18:55] expecting at this point. So, we need to really prove and to make transparent how did we come up with that match? So, not only by the way waiting, but really also by having recruiters be able to So, um to to dig down on how this uh in
[19:14] an intuitive way on how this um suggestion came to pass. And the third one would be can I trust trust the system tomorrow? So, what if data changes? There might be some reason why uh CVs are fundamentally changing, for example, by the emergence
[19:30] of new job uh types. So, imagine we have a stock of 2 million CVs, and then there's a new profession or or job label coming up being prompt engineer. How can we make sure that this is not being penalized for being a new concept
[19:46] instead of being penalized for being something somebody made up. So, this is really also what we want to see here um based on the monitoring we put in. And because I didn't express that in that on that slide already again,
[20:03] basically, we always make sure that demographic bias judges are um in place along every step of the way. Now, to talk a little bit more about the
[20:18] extraction piece, uh Sid mentioned how critical it is that we have strong uh deterministic extraction at the early stage of the pipeline, uh partly because of the audit trail and compliance associated with that, um
[20:34] and partly because we want very very strong accuracy early stage in the pipeline because there'll be multiple different iterative steps that require LLM uh computation as well and processing uh and and other deterministic steps. So, that first pass is exact string
[20:49] matching. Uh there's no cost associated with that, which is fantastic. And then we have a second pass that is a fuzzy match uh only for what's left over that didn't uh get through that first deterministic pass, uh for example, ML2 machine learning. Over time, there might
[21:06] be the ability to bring those into the deterministic checks, especially if you see that more consistently in CVs. Uh additional to that, I wanted to highlight again how important evals were for us along the process of building this. It's going to be iterative and
[21:22] continual. Build more and more evals over time. Uh that's mapped to things like precision recall F1 on a per CV basis. Um we want to look at things like location, country, region, all of that,
[21:38] and and roll that into a singular score, which is fairly complex technical challenge. Uh and then additionally, around occupation, um we need to be able to match occupations semantically. So, it's not something
[21:53] that can be done in a deterministic way necessarily, and it requires an LM that incurs cost, that incurs latency, and that all needed to be factored in as well to ensure that, you know, we met that end-to-end latency timeline.
[22:15] And then around matching and drift, um we we wanted to make sure that we adhered to a number of different evaluation mechanisms to handle demographic biases, as I mentioned. Very, very important at every step to make sure that these systems are
[22:32] fair. It's a big reason why these aren't trusted is because the initial thought that comes to mind is, well, is it going to have bias across the models that you call or across the CVs that you parse? And so, it's very important to introduce a number of different measures. You're going to see a few of them here that are
[22:47] both judges, but also done through the evals themselves to make sure that demographic demographic bias is accounted for. Uh and then we also are building in a substantial volume and more and more every day of ground truth examples, right? How should a CV be parsed? How
[23:04] should a placement be made against both common and uncommon examples? And the idea there is there's so much uh variety across the different potential matches that could be uh captured. And so, how do we make sure that there are enough eval pathways to map across all the
[23:20] different potential edge cases that might come up? Uh there are some uh you know, both the the white-collar and blue-collar as Seth mentioned earlier. It's also about frequency. How often are you matching a job? You know, how many uh times are you going to you know, match a an executive type role versus,
[23:37] you know, a someone that's doing a you know, a kitchen job for example. And so, we need to factor that into the design of a product like this as a result.
[23:54] Yeah, and then coming back to governance, cannot overstate this being from regulated land. Um what's very important for us is that the governance here is built in and not bolted on. So, basically anti anti-bias uh for both ethical but also comply to
[24:10] compliance reasons. Um but also auditability um on the one hand to be able to see if we really improve over time, but on the other hand also in order to be auditable because actually in Germany you could be sued for the fact that you cannot prove how the placement came to pass. Um
[24:28] meaning that we have to be legally consistent here. So, for example, what we are doing here is we drop certain things at ingestion already. Meaning, if we take them out at the first step, they can never be causing any bias movement. So, photos, date of birth, nationality
[24:43] never go into the system actually after um they've been extra the extraction process has happened. We have a bias judge in the eval pipeline um which which uh Chase already um elaborated on. And what we also do need to monitor is that there is no proxying
[25:00] of protected characteristics happening. So, in any case where you have an say learning algorithm even, um there might be the fact that there are say proxy signals which mask basically for example ethnic aspect.
[25:16] You could basically have a proxy for a racial bias, so to speak, which could be derived from a certain set of characteristics we still have in the system, so we need to account for that. Um we have local regulation compliance, which makes this even more fun. So, for
[25:32] example, we do not only have the GDPR in the EU, we have the Privacy Act in uh Australia, and we have bias testing mandates in Colorado and New York, for example. So, even in the US at the state level, there's variation to the regulatory compliancy requirements.
[25:47] Then, this whole thing is being characterized as high high risk AI application in the European Union, meaning this gets a lot of attention. Obviously, also by the regulatory um body, and we have human in the loop by design. Um auditability we established
[26:04] with the great product Unity Catalog. So, basically, we have can achieve lineage down to the token level in the actual CV. Um we have the inference tables being locked also using Databricks technology um for every prediction, and we use uh MLflow and a
[26:21] GDPR engine in order to have a clear model versioning as you should do also for quality reasons. Yeah. And this is where what we achieved to wrap it up. We went from 20 seconds
[26:37] latency um to below three now, and I'm very positive that we will be arriving at the sub-500 milliseconds in the second half of the year. We will be spending time together on this product. Um so, search
[26:52] la- latency, we uh are already on a good path to meet that inside uh our budget, which makes my boss happy, obviously. Um, we have one code path, which um basically makes serving an evil running on the same code, lineage from raw CV
[27:11] until served match, which is absolutely critical for us. And we are in prod, basically. So, we shipped the product in an alpha version in Australia and New Zealand just Monday. We'll be shipping it in the German-speaking countries um just until by the end of July, and we
[27:27] will be shipping it to all other English-speaking countries by the end of the calendar year. And then we will do the fun stuff, including multilingual countries with a different language logic. For example, Japan will be my personal favorite, multilingual CVs with different language
[27:43] systems. Um, so, this these will be the tough cookies. Really interesting technical challenge, yeah. All right. So, what we learned. Um, data standardization across geographies is basically the hardest mile.
[28:00] So, really standardizing job concepts across different labor markets across different countries was one of the largest challenges we faced. What comes next is we will have a graded ground truth, which uh
[28:15] promotes basically the standardization across uh geographies. So, what we will be having is basically a super ontology sitting atop of many regional or national ontologies, which makes those concepts mappable interchangeably between Hays
[28:30] countries. Keep LLMs out of the latency hot path. I'm a personal fan of that, because I'm a statistician, actually. So, I like solving problems with mathematical uh concepts from the Middle Ages. Um, so, basically, uh next what will
[28:47] come next hyperparameter optimization of the scoring levers. And then really important for us also because we do not have that non-noisy data, distinguish missing data from irrelevant data when scoring. So, in order to be able to enrich uh
[29:02] and we are taking that uh uh uh a bit prematurely, but really this is likely heading to a graph-based architecture in order to keep at those latency goals we are having. Um meaning we are really need to be um able to prevent introducing noise into that architecture
[29:19] from the get-go. Yeah. And then uh around the single code path, uh we we sort of designed this product as a platform, uh not as kind of an isolated app. You could imagine deployment of different applications for each region, different flavor of each
[29:36] pipeline, but instead we decided for a more generalizable, parameterized platform that can then be deployed with all of these different conditions handled. Uh and it's allowed us to It was a little bit of extra work at the beginning, but it reaps the benefits downstream of
[29:52] being highly scalable, uh parameterizable in the event that we encounter what will be many many unexpected challenges, of course, across the number of uh months upcoming. Um and it also allows us to do evaluation in the future across different challenger modes. So, there
[30:07] will be many different architectural decisions that need to be taken downstream. And the idea is to be able to objectively evaluate across different uh uh technology decisions across those those coming months. Uh and then ground truth, and then this
[30:23] goes for any project that that we do in AIFDE, uh and probably any AI uh responsible AI product that's being built and delivered is around the ground truth building. Takes a lot of time, especially in these cases as I mentioned earlier where there's a lot of variety,
[30:39] to be able to have a uh a rigorous enough set of ground truth, you need to bother users quite often to develop those as well. So, that's something that's going to be fundamental as we as we scale that up. And then lastly, um we want to keep the users in in that
[30:55] eval loop. There's going to be some human feedback reinforcement of this this tool over time. We want to get their feedback so that we can build a better product, build a better user experience for them. There may be be differences in markets in terms of how the recruiters operate
[31:11] and we want to make sure that their their beliefs are adhered to and then that their needs are are seen after effectively.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.