Agentic PHI De-Identification Across Multimodal Healthcare Data on Databricks

Summary

The Agentic De-Identification Framework treats the PHI de-identification pipeline as an autonomous agent, applying privacy-preserving transformations across structured data, clinical notes, imaging metadata, and PDF documents with minimal human effort using Agent Bricks and the Mosaic AI Agent framework.
De-identification must preserve linkability across records—connecting a medication event to a follow-up visit or lab result—without exposing actual patient identity, and the framework uses specialized models fitted to each customer's data distribution to meet this requirement.
AgentOps is operationalized with MLflow tracing, Unity Catalog governance for core de-identification logic registered as governed functions, and Mosaic AI Gateway Guardrails to deliver auditable, secure PHI protection at production scale.

Agentic PHI De-Identification Across Multimodal Healthcare Data on Databricks

De-identifying clinical data is essential for research and secondary use, yet current approaches are costly and require constant manual tuning as data distributions change. this video introduces an agentic approach to automating privacy-preserving pipelines on the Databricks Lakehouse.
We present an Agentic De-Identification Framework that treats the pipeline itself as an autonomous agent, minimizing human effort while maximizing security and compliance.
Attendees will learn how to use Agent Bricks to fit de-identification models and entity resolution logic to customer data; apply agentic de-identification across structured data, clinical notes, and imaging metadata using Mosaic AI Agent framework; and register core de-identification logic as governed Unity Catalog Functions.
We conclude by showing how AgentOps is operationalized with MLflow tracing, Unity Catalog governance, and Mosaic AI Gateway Guardrails to deliver auditable, secure PHI protection at scale.
Talk By: David Talby, CEO, John Snow Labs ; Veysel Kocaman, CTO, John Snow Labs ;

Chapters

FAQs

What is the Agentic De-Identification Framework presented in this video?

The Agentic De-Identification Framework treats the entire PHI de-identification pipeline as an autonomous agent that minimizes human effort while maximizing security and compliance. It applies de-identification across structured data, clinical notes, imaging metadata, and PDF documents within the Databricks Lakehouse. Agent Bricks and the Mosaic AI Agent framework are used to fit de-identification models and entity resolution logic to each customer's specific data distribution.

Why is preserving linkability important when de-identifying clinical data?

Removing patient identifiers without maintaining consistent pseudonymous keys makes it impossible to connect records across time and modalities, which defeats the purpose of secondary-use research. For example, linking a medication event to a follow-up visit or a lab result requires that the same patient be identifiable across records without exposing their actual identity. The framework maintains this linkability while satisfying regulatory de-identification standards.

Why can LLMs alone not handle production PHI de-identification?

According to this video, LLMs alone lack the regulatory-grade precision and deterministic guarantees required for PHI de-identification at production scale. Clinical data spans multiple modalities—text, images, lab reports, and PDFs—each requiring specialized models and entity resolution logic that a general-purpose LLM cannot reliably provide. The agentic framework combines specialized de-identification models with LLMs and guardrails through Mosaic AI Gateway to meet compliance requirements.

How does Unity Catalog support governance in the de-identification pipeline?

Core de-identification logic is registered as governed Unity Catalog functions, enabling consistent policy enforcement and access control across all downstream uses of the pipeline. Unity Catalog also supports audit trails and lineage tracking so that organizations can demonstrate regulatory compliance. Combined with MLflow tracing, this gives healthcare organizations full visibility into how PHI was handled throughout the pipeline.

Full transcript

[00:09] Secondary use is everything else. Uh so for example, if I want to look at all the people in my hospital over 2 years, and say, "Okay, you know, how were they doing with long COVID?" Okay, so what symptoms are we still seeing 2 years later? Right? Or we we changed the way we do a specific heart surgery 3 years ago from,
[00:22] you know, this type of stent to this type of stents. What are the recoveries rates, quality of life, right, for all of these people? If I'm designing a clinical trial, say, "Oh, I want late-stage breast cancer,
[00:34] uh you know, survivors, right, with these specific biomarkers, for example." Uh for this, I need to work on a very large number of people. Okay, usually I'm looking at people over multiple years, multiple modalities. We
[00:46] have a lot of information about them, but what I do not need is their names. Okay, because I'm not going to go to them and do something specific. I'm doing research about what's already happened, what's already documented. Uh so, really de-identification is
[00:58] something that we see now really basically every healthcare system, every payer, everyone who has access to this kind of data. One of the things we want to do, I wanted to de-identify data set because there are so many use cases I'm running just on that.
[01:11] Uh so, de-identification is a very common request. Uh it's also very hard to do. Uh because when you de-identify, you of course it's very hard to de-identify so that you cannot do anything with the data. Okay, so for example, what you do not do
[01:24] is you do not take all the the names and and patient identifiers, you just remove them. Okay, that's the last thing you want to do because then if I have let's say I had a hospital visit here and then a radiology visit here and a lab here, you can't
[01:36] even connect them. Right? So, you can't ask super simple questions like, uh you know, "After we gave you the medication, did you get better?" Right? Or I want to see, "Oh, side effects side effects also they happen after you take it, right?" Or I want to
[01:49] know that you actually came back and did your physical therapy after you did the surgery. Uh so, I need to be able to tie things over time. I need to be able to tie data over different modalities, right? This could be a PDF, this could be a you know, an image, this could be a lab
[02:01] report, this could be just text of something that happened. Uh and I don't want basically to over redact. Uh the other thing I want to do is I want to at least do this lawfully.
[02:13] Okay, so there's the the the basic part which is yes, when you say that something is de-identified, it's a legal definition. Okay, so at the end basically we show you uh how we do part of the expert determination letter, but the end of the letter you give it to your legal team
[02:25] and say, "Okay, now you can share it with this pharma company or with these researchers." Um so that's a requirement, but that's usually the baseline and usually you have to do better than that. Okay, whether you're in the US, but even if you're in the EU, um
[02:39] the legal requirement is usually slightly below what you need to actually sleep well at night and we will see some examples of that. Uh so we need to look at that. And uh yeah, we talked about the fact you need to be having to be able to do it across data modalities.
[02:51] Okay, so for most uh research today I want to look at it uh free text, I want to look at structured data, I want to look at labs, I want to look at images and look at them together, right? So if we're studying what happened to the tumor, I want to see the images, I want
[03:03] to see the radiology reports, I want to see the lab results and I want to see the actual structured data, right? What medications, treatment paths, uh paths, all of that. Uh so this is how it looks like on text.
[03:15] Uh you have on the left you have the the original data and the first thing you want to do is just identify personal elements, right? So you know, names, identifiers, dates, addresses, uh
[03:27] uh personal information. And then one thing, when you de-identify, you do not just remove the information. Uh usually you do you configure what you want to do per field. Uh so for example, here on dates we did date shifting.
[03:40] Okay, so you don't want to just generalize all dates to a year. Okay, but it's actually illegal to just leave exact dates. So usually what you do is pick a random number. Here we picked eight. And then we say, "Okay, shift all dates
[03:52] consistently by this number of dates." And then all the dates are random because the number really the the number is random and hidden. Uh but you still all the relative dates work correctly. Okay, so you know what happened before what, what happened within the time
[04:04] frame. Uh identifiers usually wants to do some kind of a cryptographic hash. Okay, or some kind of tokenization so it's irreversible but it's still consistent. Uh addresses you want as you can see to obfuscate, names you want to obfuscate.
[04:18] Right? Meaning you just invent a new random name uh but then you have to use it consistently. Uh so that's that that is basically the job. And of course you have to do it across modalities. Uh so if you have for example a PDF uh and PDFs could be digital or it could
[04:31] be visual documents uh you want to make sure that you can uh either redact or do the obfuscation on the PDF itself. Uh the only additional challenge there is that you have things like font size
[04:44] and font. Right? So if you want to draw a rectangle rectangle in right coordinates, you need to know really uh that it covers the word correctly and not nothing beyond it. Uh when you look at medical imaging uh like here you see on the right uh what
[04:57] you have sometimes is you have PHI burned into the pixels. Okay, it's not on it it is in metadata but not only the metadata. Uh so if you have the patient name appearing on the chest x-ray, you have to remove it. Uh but what you cannot do is
[05:09] you cannot just remove all text. Okay, because uh sometimes for example uh laterality, right? So you take the picture from the right or from the left is super critical. If you remove it, basically the image is useless to the radiologist.
[05:21] Okay, so you have to actually understand what each item means and decide what to do. So here are some of the things uh we we have learned over the years on how to do these things correctly. Uh first of all and most importantly
[05:34] is that with today's technology, it is very very possible uh to automate uh I'm sorry, to completely automate uh regulatory grade clinical data de-identification.
[05:46] Okay, so about uh 5 6 years ago, you would still have to do it with people, right? So, you you you'd run kind of an AI assistant or an ML assistant thing, and then you'd have humans review it. Uh today, you really
[05:58] do not need to do that. Actually, what we find if you ask humans to review it, you introduce more errors. Okay? Um which means that a lot of the uh healthcare ecosystem is now moving and moving fairly quickly
[06:10] from de-identification, okay, if you do a study, you need 5,000 patients, let us know, we'll have a team, they'll do it manually, to no, no, no. For example, here with Providence Health, and you see we have uh we wrote a couple of papers with them.
[06:22] Um they took 2 billion clinical notes, their entire history. Let's de-identify the whole thing in advance. So, the people can just go and do research. Right? And once data is de-identified, you do not need to go through an IRB or
[06:34] human subjects review because there are no human subjects. Right? You're just looking at the population as a whole, uh meaning that you can start your in practice, you can start your project usually 9 to 12 months earlier. Okay? Because that's how long it takes.
[06:47] Um with Providence Health, we we've done a few other things. Uh they've actually done uh an evaluation versus their own compliance team, right? People went and manually reviewed and said yes, it was over 99% accurate.
[06:59] Uh we also did a a bias evaluation. Uh so, some of the things you want to make sure don't happen to you is, for example, that you do really well if the patient's name is Michael, but if the patient name is, you know, Wei,
[07:11] right? Or Muhammad, right? Then you're not as good. So, you need to make sure that uh you're looking at people's uh race, religion, country of origin, right? And you you don't have biases
[07:23] uh this way. Uh so, we've done that work and that analysis and optimized the models. The last thing they did, they they actually brought in a consultant, they brought in Deloitte for 3 months, and they said here's a set of a few hundred patients, uh go just do right to me, go
[07:36] re-identify them. Find out who these people are. Okay? By any means necessary. And after 3 months, uh they came back with one name, and that name was a fake name. It was an obfuscated name, so really they were not able to find any patients back.
[07:49] And then they said, "Okay, let's take this to production." Another thing we've learned is that you really do not want to do the identification with an LLM. Okay, because today some people say,
[08:01] "Okay, you just, you know, prompt the LLM and you, you know, configure a prompt really well, and then you tell it to find those things, and it finds them." And what you find is that that that is really not how you want to do it for two reasons. The first one is accuracy.
[08:14] What's impressive with LLM is they do this really well with prompting only out of the box, and you get to low 90s in terms of accuracy. Okay, the challenge is with the identification is because you need to be legally de-identified, low 90s means
[08:28] absolutely nothing. Right? It's not something you can take to to a legal team and tell them, "Yes, this data is legally de-identified." Okay, which means that, you know, it's the same. You know, you might as well do it with rules right, and then you're saying to your team to review.
[08:41] In order for data to be considered legally de-identified, you need accuracy to be usually around, you know, 98, 99%. Or more in a, you know, provably representative way.
[08:53] And LLMs just they just do not do that. And you can see this kind of You can see the preview paper with a lot of my hand waving about, you know, how how we actually measured, you know, what's the data set, what are the metrics is explained in the paper.
[09:06] Which is why we sent it for peer review. But that's one thing. The other issue is cost itself. Okay, you cannot do this and pay per token. Okay, because the identification is very expensive. If you want to de-identify
[09:18] one patient, one patient is say, it's, you know, 1 to 1.5 million tokens on average. And you need to to de-identify you need to send the whole thing in, and you need to get the whole thing out. Okay, de-identified. And that's
[09:31] excluding, you know, your prompting. It could easily be like several dollar dollars per patient just to do that. Okay, which is completely ridiculous. Right? If you have, you know, hundreds of thousands or even a million patients. Uh so, this is not an option. Uh
[09:44] therefore, the the way we do it, the way most of the industry does it, is what we do. We build small task-specific models. By small, I mean like 10 million parameters, not billion.
[09:56] Uh that find those things. We put them into pipeline. And a lot of what we sell will talk about is how we configure and optimize those pipelines for different very specific customer use cases. Uh and they run locally. Okay? So, there's no per token cost.
[10:09] Everything runs within your environment. Another very important constraint is you should not send your PHI somewhere to an API to get it de-identified. That kind of that beats the whole purpose of the fact that you're not supposed to ever send it anywhere.
[10:21] So, really what you want, you want to have small models that you just take the software, you install it where the data already is. Right? So, for example, if on Databricks, this runs on it just runs within your VPC, within your account.
[10:33] No air gap, no external API call, no nothing. Uh you want it small so it can run optimally at scale. And then the cost works as well.
[10:47] Um another thing we found is that same thing. There's also so peer-reviewed comparisons versus things like, you know, Amazon Comprehend Medical, Azure Text Analytics for Health, GCP models. Uh also generally, if you use an API that's out of the box, you're going to get lower accuracy.
[10:59] And if you fall if you fall below you know, 95, 96% accuracy, basically you're just not solving the use case. And these APIs are are usually far far below that, right? Usually, you're lucky if you get to 90.
[11:12] Um and part of it is really just how much kind of, you know, training and and specificity went into it. Partly, are things that you need to customize. Right? For very specific use cases, which is the the problem that we want to
[11:24] talk to you about today. So, in order to reach to cross the 98% channel, usually what you need, uh you don't just use an off-the-shelf thing. You come, you start with the off-the-shelf, and then you start fixing
[11:37] and optimizing it. Okay, you may need to fine-tune a model to a data set. You may need to add specific rules. You may need to order things, right? To say, "Okay, I'm doing this data pre-processing, then I'm finding the entities this way." Something is I do just rule-based, you
[11:50] know, like emails, right? Or license plate, those kind of things. I decide in what order I do them. Uh and then I I keep iterating. This is something that can take a team of people who who know the code base, who know health care, you know,
[12:02] from you know, from 1 to 2 weeks to could be, you know, much more than that, depending on how complex the data is, how many modalities. Uh and right now there are we we actually map this because we automate this process. There are 95 different
[12:15] knobs and decisions that, you know, you can pull and tweak and turn in in how that happens until you get to it. So, it's a highly iterative process. What you build, you build a hybrid pipeline, right? Some small language models, uh some deterministic one, some rules,
[12:29] uh uh and how you how you put them together in one pipeline, right? So, that the text runs and that happens. Uh and the project have the project can get fairly complex.
[12:41] Uh the the first thing that get complex for us usually is just the sampling piece. Uh if we're lucky, we actually have access to the data and we can do uh the sampling ourselves ourselves. Sometimes you need to actually tell people what
[12:54] you need because as you know, models will not work if the the sample data that you annotated or you trained on this not actually represent the real patients. Okay? And it's not as easy as that. It's not give me people with a good distribution of ages and good
[13:06] distribution of uh you know, gender. It's oh, I need to have, you know, enough diabetics and enough hypertensive hypertensives and enough cancer patients, right? Enough people with breathing difficulties, right? And it comes to, you know, 20 or 30 different
[13:19] dimensions that you need to get kind of just right. Otherwise, uh clearly look, you know, in in US older populations like, you know, diabetics is like, you know, a third of the population. Hypertension is more than half of the population.
[13:31] Not having the right distribution of hypertension is like not having, you know, males at all. Okay? So, these are not, you know, 1 or 2% parts of distribution. These are things that will completely kill your model in a very obvious way. Right?
[13:43] Because they're, you know, major major uh parts of the population that that have these conditions that you need to account for. Um so, you have the sampling. You want a baseline. Uh we we do have some annotation, right? You can say, "Okay,
[13:55] how far off are we? Uh we're you know, when when real doctors go and annotate the data on a small sample." We look at the gaps, and then we iteratively work to improve the gap. Okay? Uh and that's that's where the the time
[14:07] and the money goes, right? If you need a highly skilled uh clinicians, data scientists who know this problem, uh who understand the domain and understand the technology, right? To iteratively understand, "Okay, oh, we we see this kind of problem. Okay, we
[14:19] understand this pattern. This is a pattern we solve, you know, with the model. This other thing we solve with this very specialized rule. Uh this one we solve with For example, one of the interesting things that happen in de-identification, uh you can have um disease names or procedure names that
[14:31] are like last names. Right? So, like, Alzheimer's actually named after Dr. Alzheimer, right? Uh doctors uh love naming surgical procedures after themselves. It's a thing. There are hundreds of these. Okay? Uh so, you know, so so, you can
[14:43] say, "Okay, oh, the model seems to be mistaken, you know, these two specific things because this hospital does a lot of that specific procedure. Okay, how do we So so, you may say, "Oh, we run, you know, we run the the procedure detection model, and then the name detection
[14:55] model. If it conflicted with this threshold, we pick this or the other." Right? So, yeah, so those are kind of the the kind of things that that you play with. Uh but at the end, uh this is really where this thing becomes expensive, but these are the things you really have to
[15:07] do. Right? And there's a legal question there. You cannot, uh you know, you cannot keep people's real last names and and real diseases in the data set. Uh what we are going to show you now is uh what we've done to automate this process.
[15:20] Okay? And this is where the the Agent workflow comes in. So, we we don't use the agent or use LLM to do the identification, right? That's just not not the There's no benefit in that. That's not the right way to do it. Um you want to use really a small,
[15:33] highly specialized model that are highly scalable, very cheap to do this work. What we use the agentic AI pipeline for is to build that pipeline. Okay? Right? So, basically what we're doing here, we're replacing the work of these
[15:45] highly specialized clinician data scientists and all the decisions they make and all the domain expert that is they've built over the years. To come and say, "Okay, look, we have the default pipeline. Now, look at the real data set. Do the sampling. Run this. See what
[15:57] happens. Run the default. Look at the errors. Reason. Figure out what's going wrong. Decide how you want to fix certain things. Run the iterations until at the end we build the pipeline that works specifically, right, for this
[16:09] scenario and this data set. And with that, I'll hand off to Basel to show you how it works. >>
[16:29] >> Hello everyone. So, as David explained, so we are going to automate this flow that we have been doing manually for the last 10 years during more than maybe 100 different project that we run in in real world data set so far. So, the goal is not
[16:41] using the agents to do this identification process. How to automate this process as if like human data scientists just like we jump into a customer in front and then try to automate everything end to end, right?
[16:53] So, what we usually do is that I will explain at first how we usually do this end to end. So, there is a If we are lucky, there is a gold annotation. Most of the time there is a gold annotation as a test set to start with. If this is
[17:06] not like a 6-month later drift detection stuff, but usually we have some golden data set that we start with to see how well we fit into their pipeline right now. So, we need to do some
[17:18] gap analysis. So, why in the second one I call large pipeline? Large pipeline is like a pipeline that we built is like 40 plus different stages that is running in a Spark cluster. Uh
[17:30] combination of several small machine learning models plus rule based stages and one more transformer based stages. We call it zero shot pipeline. Why why it is there? It is basically there if I make an analogy, it's it's there to
[17:42] vaccinate the rest of the stages. Because at the end in production, we plan not to use it in production because even if it's very small like half billion size of model uh it's an it's a deberta base uh transformer pipeline,
[17:57] but it is still very hard to scale even on a GPU because it's we we are talking about billion level of data. So, and then the third row we we are talking about optimized pipeline. Optimized pipeline is just a stripped
[18:12] off this transformer base modules from the main pipeline. Uh that is working really fast on your own infra with no GPU needed. So, right so that you can then you can
[18:25] identify 1,000 notes in in 20 seconds in a 64 core machine. This These are the benchmarks all coming from these experiments, by the way. So, at the end we adapt this pipeline. Why do I call it pipeline? Because this
[18:38] is on top of Spark NLP and Healthcare NLP library uh that needs to run without any pre-processing or post-processing, right? So, if you write something in Python and then try to do some post-processing afterwards, it will not
[18:51] scale when you try to ship this if you are not, you know, putting everything into a UDF functions, etc. So, that's why at the end we have one adapted shipped pipeline that needs to catch the recall above
[19:03] 98%. So. So, those are the components that we have in the pipelines. So, by the way, when you see these pipelines, these are all usually one-liners right now because
[19:15] we stack them and then like we we we we we club them together and then upload the model up so that any in any collab notebook or or database notebook, you can just pull it in as a one-line pre-training pipeline. So, the large
[19:28] pipeline, which has zero-shot inside, let's say that it's right now on the that customer golden test set during the gap analysis, this is recall is 92. So, optimized pipeline is this 88, which
[19:41] means that it is 4% worse, right? So, but at the end, we have an adapted pipeline that needs to achieve about 97-98%. So, as you see, the adapted pipeline is a different representation of the
[19:54] optimized pipeline that has no heavy stages inside, but all tuned on the customer like your own data. We have one more additional named entity recognition stage over that and a few other rule-based stages because
[20:07] when we start, there are not everything that you see in the data as a PHI should be extracted via predictive named entity recognition models, right? Because there are some rules that can be easily ruled out like some date patterns, zip
[20:20] patterns, etc. You don't even need to increase the model capacity or try to increase the model capacity just to catch something that is already well-defined that you can just write a few regex, right? That's why By the way,
[20:34] our experiments also say that even with the rule-based stages, sometimes we achieve like uh 74% uh PHI recall uh on some data uh set, but when the data
[20:46] starts shifting, uh and then you have some you go to tails of your data, right? In your distribution, of course, it will not be enough. That's why we have all these checks and balances. And while doing this,
[20:59] we usually make too many decisions, right? So, again, this is the representations of the pipeline in your data. We try to achieve that upper line so that optimized pipeline can achieve
[21:11] the expected level of accuracy on your own data by covering because if you just use the large pipeline or optimized pipeline as it is, you will see that it's not going to hit all the
[21:23] all the slots that you would like to cover, right? So, that's why we try to build a similar like a typical version of that pipeline with your own data with the same level of uh
[21:36] expensiveness. So, why so many stages? As I said, this is like 40 plus sometimes stages that are over there which you just pull with one liner in your own notebook. But,
[21:49] not everything can be ruled out. Not everything can be used to train a model, right? Because if the name is weak over there, if you keep adding more names, you add other you introduce other entities to the model as
[22:03] well. And then, you need to the model needs to have more parameters to cover all this discrepancies, right? But, we don't have that budget right now. Because this is a
[22:15] de-identification, usually this is working behind firewalls. Sometimes, like most of the time we don't even as a data scientist working with these projects, we don't even get to see the data sometimes. So, we need to work with
[22:27] eyes closed totally. And then, we usually build the pipelines without seeing the actual data. But, the data scientist or the the engineers working with their own data, of course they work
[22:39] with this, but they don't have they may not have the same level of experience or years to, you know, which not to tune, right? We are talking about more than 90
[22:51] different decisions that they need to make to make sure that their data is well obfuscated at the end of the day. That's why the combination of multiple stages at several machine learning small machine learning models and
[23:06] a couple of rule based stages out there. And each one returns something from your own like high less something from your own data. So how it is done by hand? When we start a project we jump into a project, we get together with them and then we start
[23:19] checking the gap at first. Uh we we like by by getting the distribution of their data some metadata some demographic distributions etc. So we sample some data
[23:31] and then we try to uh we we annotate it at first. We have human physicians who do the annotation. I'm explaining how the projects are usually done. Like like the projects are usually 45 weeks. All right, so we have some scope over
[23:44] there to make sure that at the end of the project we are able to deliver a pipeline that they can ship into production. Uh when I say they this could be we as well, right? So I am like with my own
[23:57] team I we start working on a data that we see that we own let's say and we we we write some ontology according some some guidelines annotation guidelines and then now we have a pipeline that says that it is
[24:10] doing really well according to papers, right? Which is GSL pipeline let's say. So we do the gap analysis over there and during the gap analysis we see all the we see the range that we need to close at the end of the day. Then we we know
[24:23] that this is the best pipeline out there let's say, but we cannot run this just like you cannot dump all your data into uh frontier relevance, right? Even if they do well, of course every other data are
[24:35] doing much better, right? We are talking about billion level of billion uh tokens or billion documents that you need to dump into again and again. And everyone is really right now using like
[24:48] we are talking about token consumption, right? So this is it's probably bankrupt if you try to do this. And by the way, we even tried to use LLM to do this vaccination rather than zero shot. What if rather than
[25:00] annotate by human, what if we use LLMs like frontier relevance uh to annotate my own documents just by 500 sample or 100 more sample to iteratively improve and then train small
[25:12] models like a distillation, right? We tried this as well, but we usually ended up uh introducing new noise. That's why we follow a different approach. So, we find the pattern at first, we find the gap at first
[25:25] uh between like the gap between actual ground truth versus large versus optimized which needs to go into production. Then uh then we annotate 500 documents. We usually repeat, we start uh
[25:38] because it's it takes like a 1 week to annotate by one human, one physician. That's why we start with 500 document. And then we train a small named entity recognition model with all this PHI. Why? Because now we uh
[25:51] we have an optimized pipeline to improve further. And then we know that how far it can go by looking at the large pipeline, right? Then we train a new NER model and then we see that when like
[26:03] old machine learning, right? We do hyperparameter optimization as well. There are many things that we can do like a dropout rate, learning rate, batch size, epoch number, etc. That we need to make all these decisions just like we have been doing for the last 10
[26:16] years, right? Then some of the stages cannot be covered by this model because these models are small. They are like 14 MB. Like it's like 10 million size of 10 million parameter size I'm talking
[26:28] about. And then the beauty of this is that they are using BiLSTM-CNN-Char architecture. They are more deterministic than the large parameter models, and then you can chain them together and then merge so that you can
[26:41] control how to merge. Like which signals coming from which stage can pass through the gate. Uh so, we have some uh mergers in the pipeline as well. Then we write some rules to cover the weaknesses
[26:54] rather than pulling 100 more data, more document, we just write a few rules to cover some patterns that we don't need to uh spend time on training
[27:06] or annotating more data, right? And then by iteratively we do this for weeks and then at the end of today we have some pipeline that can run in production uh till we have a data drift. Then after
[27:18] a few months you detect some data drift and then you keep you you you you start from almost scratch and then you try to fix this again and again. That's why we said that what if like since we are doing this right now, we
[27:30] are able to do this right now by iteratively talking to each other, seeing the data, plug and play, etc. Why not we automate this end to end by building the pipeline piece, not doing the actual identification. All these
[27:42] decisions that we need to take uh during building this production-ready pipeline, how can we automate this flow? So, we built some self-serving loop, self-healing loop,
[27:54] uh and then we kind of relied on even if there is no gold annotation, we said that what if we use the weak annotations like a self-annotations coming from pre-trained pipeline, which is let's say 85-90% accurate.
[28:07] Since I know that in the production pipeline I have small models, when I train small models, the beauty of them are they are over uh they are uh trying to find the pattern, right? They are ignoring the noise in your own
[28:19] data. So, even if there are some label errors, we thought, "Okay, what if I since I have an infinite number of raw data, let's say. I don't need an annotation. If I have an annotation, of course I can
[28:31] have a hybrid version of this as well." Uh For instance, there are some large-scale projects right now we are running that we haven't seen the data for the last 1 year. And we are still trying to improve. Why?
[28:43] And then with this one uh with this, you know, the self-healing loop, what we did over there is that we pre-annotate all, let's say, 1,000 documents with the large pipeline,
[28:56] because we have a budget for running that large pipeline, right? Because we are talking about 1,000, and this is not even a production. We are just trying to find some subset to start with. We at first stratify by metadata, and then we
[29:08] pre-annotate. But, there are some noise introduced to the pipeline, right? To the pre-annotations, right? Because these are all weak labels we are talking about. So, we add some self-healing loop over there. Self-healing loop is
[29:21] it's like again the video small models. If you train a small machine learning model on a data set, and then try to get the I'm I'm going to say something weird, by the way. Get the prediction on the same training set as
[29:34] well. What happens? So, you see that the model has seen this, and it's a small model. It should have predicted well, right? If you didn't accidentally overfit, of course. That's why you like if I
[29:46] uh, for instance, if I train 50 epoch, it will probably overfit, and it this technique will not help me. But, if I train it well, and then and then get a prediction, everything that I have seen so far as a model
[29:58] uh, that I'm not able to predict well, they are probably labeling error as well. By doing these kind of techniques, uh, some threshold gateways, etc., we try to do some self-healing over there before start training a new model with
[30:12] this uh, data, pre-annotated data. I'm not going to put this into production, of course. I'm going to do this iteratively, so that we heal this pipeline again and again. Then, I already have three to four
[30:25] different NER models inside. I have five plus different stages, like rule-based stages inside. Then, I plug back in this new NER model. I merge them,
[30:37] and now I have a new pipeline that I can keep doing this again and again, right? There are some, of course, new rules that I can add or change some of the rules, which I'm not able to do without LLM, right? That is the only piece
[30:50] because this pipeline entirely works without LLM as well. Because they are all gated. But only thing that we rely on, if there is an LLM mode activated, writing some regexes just like we do
[31:03] some because they are really good at writing codes, right? So, they can find the pattern on the training data by getting the false prediction like a self-prediction. If Yeah, if I feed the false positives through self-prediction and then ask LLM
[31:17] to find the pattern over there, what do you see over there? Why I am keeping failing on this? It's writing It's probably going to find No, that's probably It's That's why it's adding a few stages and dropping some of the stages, which is helping really well, by
[31:29] the way. Then we have the gates for each entity, like recall 98 for each. Sometimes, if you try to It's like multi-criteria optimization, right? If you try to hit 95 on each one of them,
[31:42] we need to make some decisions over there, right? When to keep pre-annotate with new data or adding new rules? We need to make some decisions. If I add more data for the weak age entities or
[31:55] contact information entities, like contacts are easy usually, right? But if it is name, it's hard. Uh so, that's why we let the model this during this pipeline
[32:07] production stage, we let the model make these all these decisions. So, I'm going to show you some live demo how it's going how it is working, actually. Uh maybe I will need to skip some of the
[32:19] slides to make it fit into 40 minutes. So, as you see, this is like this loop it uh iterates further and then we try to fill the gap on the left-hand side so that some of the red predictions turn to
[32:32] green. So, we have we are right now creating our own production pipeline. Uh and then we try to hit all the uh gates during this process.
[32:45] If it is not enough, since we have more data right now that I can keep re-annotating with the same the latest pipeline again and again. Right? I don't even need to pick the first large pipeline. I can keep picking my latest child pipeline again and again. Then I
[32:58] ship this. Then there's a drift. Like after 6 months or 3 months later, there are some three drift triggers that can be triggered and then everything starts to fall apart. But if this loop is active over there,
[33:11] it can start the healing process again. Right? And then you have a new pipeline. New production pipeline again and again. So and then at the end of the project, we usually ship expert determination
[33:24] document to explain the risk if they ship this so that they can sleep well during the nights, right? And then usually human baseline is around because most of the time when we try to improve by using the human annotations, we
[33:36] notice that most of the error comes from the human as well. So just like I do self prediction, if we ask the same physician a second time about the same entity, they usually change their mind, right?
[33:49] And then we do this all the statistical uh analysis before we ship the pipeline or before they ship their pipeline into production. So before a manual project now became a self-adapting automatically
[34:01] uh self-healing pipeline which is which the which is built on top of an agentic workflow to build that pipeline, not building the not running the pipeline or being inside the pipeline.
[34:14] Then so we can we can mention like 10 different agents over there. Each one of them is making different decisions while building this pipeline. And then even the expert termination writing, writing all the
[34:27] statistical tests are there that can be handled via agent. And we have LLM and no LLM models well, which means that you don't need to have LLM if you have LLM. Of course, you can add that rule-based stages as well, which
[34:40] might be faster to finish. But, if you don't have, it is still highly gated pipeline, which can run in production. Because sometimes the model we need to make a decision to stop training, right?
[34:54] And then these decisions are all can be done by the LLM over there as well. So, no LLM core version, we have two cores over there. And then LLMs help especially for the rule-based
[35:06] stages. So, one framework, this framework can be applied all the other modalities that they will also cover, right? So, we have structured data, unstructured, DICOM, CDA,
[35:18] PDF, etc. So, the same method can be applied. Of course, we start focusing with the unstructured text as first, and the others now we are trying to integrate the same into other modalities.
[35:30] Now, whatever we started with 95%, let's say, now um after gold audit, sometimes we even see that the test data set has some noise as well, even with the golden annotations.
[35:42] If we audit it, we sometimes even hit 99% accuracy over there. And uh whatever otherwise, if we didn't have this, we we would stuck at 88 88 recall as you see in this deck, right?
[35:56] So, some just with the rules as you see on the second column, uh even with the rules, you can achieve like about 90 about 70% accuracy, by the way, on some entities. But, if you are talking about
[36:09] names, locations, etc., you may never be able to do this with the rules, right? So, how this all maps into Databricks infra? Right now, the agentic workflow is a plane like built plane, which we are
[36:23] using uh which we are uh all the all the benefits from LLM's, making all these decisions, otherwise I would have I would have done. So, uh all these lay all these components
[36:35] can be run can be fit into one Databricks stack, right? So, I can use I can have the adaptation controller on the Mosaic AI agent framework or I can deploy my LLM via Mosaic AI gateway as
[36:47] well. And then, since this is a there's a harness as you see over there that this like all these decisions needs to make just like Cloud Code itself, right? So,
[36:59] this this harness this framework can also be improved as well. Right now, we coded this. But, there are some agentic harness optimization tools as well, which Databricks also has. Uh
[37:11] for instance, agent bricks. We can also deploy this over there and then try to improve with your own learnings, by the way. Because whatever you do, you you can even add some human in the loop as in a
[37:23] dashboard. In our tool, we have it, by the way. I randomly pick, select, okay, I reject, approve, and then the next time it learns from these interactions. And then the run plane, it also runs on your lakehouse, right? And then one
[37:36] single pipeline that can scale immensely using Spark clusters. And then you can even detect the drift. And the final production pipeline can be registered to a to your to a Unity catalog as a
[37:50] function. So, that it can be triggered whenever there's a drift detection. Right? Uh and every stage in production version stays runs on the
[38:03] lakehouse. The healing doesn't even need to, you know, use all that bunch of data. Uh so, you the production runs on lakehouse. And all almost everything scales. Otherwise, you would need to
[38:15] babysit right rest. And this is a one Python module at the end of the day, by the way. This is shipped, so this is one binary right now that pre-trained pipeline. Now, you can run this pipeline production or pull all
[38:28] these pipelines via as I said more than 90 different parameters. We even try to use LLM to decide because I I kept adding new arguments again and again, new parameters, and then at the
[38:41] end of today, I wasn't able to decide which one to pull, which level to pull. Then I said, "Okay, let's ask LLM to decide which one to pull again and again." So, and then it did the job.
[38:53] So, the yesterday what was single project now become a pipeline adapts itself. So, that we faster on boarding and auditable. Usually, we spend time just
[39:06] for auditing or there are some project I spend 1 month just for sampling, just to convince them about sampling because they don't allow me to see the data. And then I explain again and again, then at the end after 1
[39:18] month, we have some sample data to start with. I hope it helps. If you have any question, please connect us. You can email us or you can connect us at LinkedIn. I hope you enjoy. Thank you.
[39:31] >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.