Solving Health AI’s Data Problem

Summary

• Healthcare data fragmentation: Understand why fragmented data is the biggest barrier to deploying AI in healthcare, payers, and providers.
• Datavant partnership: Learn how Datavant leverages Databricks to centralize clinical data from the largest health data network.
• 80% efficiency gain: See how standardization cuts AI deployment work by 80% while preserving data integrity.

Solving Health AI’s Data Problem

Talk by AJ Aitken, SVP Infrastructure & Platform Eng., Datavant
AI in healthcare has a data problem. Fragmented data remains one of the biggest challenges, and bottlenecks the development and deployment of AI solutions across life sciences, payers, and providers. Legacy paper-driven workflows and fragmented technology perpetuate silos, making it difficult to create a comprehensive, real-time picture of patient health. Datavant is leveraging Databricks and AWS technology to solve this problem at scale. Through our partnership with Databricks, we are centralizing storage of clinical data from what is arguably the largest health data network so that we can transform it into structured, AI-ready data – and shave off 80 percent of the work of deploying a new AI use case. Learn how we are handling the complexity of this effort while preserving the integrity of source data. We’ll also share early use cases now available to our healthcare customers.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is Datavant's role in healthcare data?

Datavant is the largest health data connector in the United States, with access to over 70,000 hospitals and clinics and partnerships with major pharmaceutical companies. This video explains how Datavant processes massive volumes of healthcare data daily to power analytics and AI solutions across the healthcare ecosystem.

Why is fragmented data a problem for health AI?

This video describes how legacy paper-driven workflows and fragmented technology perpetuate data silos, making it difficult to create a real-time picture of patient health. This fragmentation forces organizations to spend significant effort preparing data before they can develop AI solutions.

How does Datavant use Databricks to solve this problem?

This video demonstrates how Datavant leverages the Databricks Data and AI platform to centralize clinical data storage and transform fragmented data into structured, AI-ready datasets. The platform handles the complexity while preserving source data integrity and regulatory compliance.

What does 80% efficiency gain mean in this context?

By centralizing data on Databricks, organizations can eliminate 80% of the manual work required to deploy a new AI use case. This efficiency gain comes from standardized data preparation, consistent governance, and reusable infrastructure, enabling healthcare organizations to focus on clinical outcomes instead of data plumbing.

Full transcript

All right, I think we're good to get started. Can everyone hear me? Yep. Fantastic. All right, these are a little strange, but let's go. Uh, thank you everyone for coming out. Hope everyone's having a fantastic conference here. Uh, probably only a few hours left, so hope you're making the most of it. My name is AJ Eken. I'm here today from data vant. I'm going to speak a little bit around uh what we've been working on over the last few years, how we've been trying to solve some of the challenges that we face within the healthcare industry and what we're really leveraging the data bricks platform for to really be able to innovate uh develop some of more AI technology and make the data more accessible for both ourselves internally as well as our customers.
So, who is datavan? Datavan is the largest data uh connector in the United States. We've got access to over 70,000 hospitals and clinics. Uh partnerships with 18 of the top 20 farmer companies. You can see here a number of the stats that we have. I would actually preface this to say that these are under reportported right now. I think as of this morning we actually made an announcement of an acquisition that we're making a company called Antelus which is also in the chart retrieval industry.
So these numbers here are going to be quite a bit larger as we look towards integrating them. Uh other stats we have we do over 40 million patients coding a year. Uh this is coding for retrospective risk of Medicare advantage patients. Uh our tokenization software in the life sciences industry. We're tokenizing over a trillion records of healthcare data annually. So this is all to say that our reach and our scale is nearly beyond most other the health care systems within the United States.
The amount of data that we're receiving daily and we're processing through our pipelines uh is excessive and this is why we've had to develop a platform that enables us to really process that data at scale to drive the analytics and really unlock the AI into it. This again is just a little bit talking about our network. Uh the data that we're receiving is primarily from our payer and provider customers. Uh there's a number of different use cases we have within data van.
There is really there was three verticals. As of today, there's really four verticals. We've got our payer vertical, provider, and life sciences. And with the acquisition of Intel, we're creating a new vertical which deals directly with the legal side, uh gathering charts for attorneys, etc. So they're the different verticals. They're the different lines of businesses that we're supporting. Datavan itself, a lot of what we're doing underneath the hood is the routing uh the routing and the gathering of charts.
We know where to go to find these charts. We know which providers we need to go talk to when we need to gather them. Uh we know how to deliver them to our customers. So that is in large part of what's happening under the hood. The data that we're gathering needs to be stored centrally in a location that's easily accessible for all of the applications. Uh doing that provides ease of access uh for all of our customers.
So let's start with the challenge that we have with the data that we're getting. Uh I would say that a number of you here are probably in the healthcare space and realize that over 80% of the data that we're looking at right now is unstructured. That is to say that this data is coming through in PDF forms. It's faxes. uh it's scanned notes, it's handwritten notes that unstructured makes it very difficult or it makes it more difficult to develop AI models or do analytics on top of natively.
What we try and do is extract structured information from that unstructured to make that data more accessible for our models. Uh other issues that we're dealing with is the silos of information. We're dealing with payers, providers, and life sciences who historically don't have direct connections between the two of them. that data is siloed within their organizations. Even within our organization it ourselves we were finding that data sitting in various silos around the place that just provided extra complexities for us as we were trying to really unlock and give uh the full full access to that data.
Uh we obviously all know about legacy workflows in the healthcare industry. Uh and AI needs structure. Obviously today with the advancements of LLMs and things like that, we don't need to have fully structured data. LLMs are still an area that we're looking into to see how we can make best use of them in the in the healthcare space. We obviously have to be concerned with things like hallucinations of those LLM. But for us, what we're trying to do is we're trying to create the best data set possible to really provide the models that are built on top of that uh the most accuracy.
So the way that we're going about that is really trying to take that unstructured data and convert it into structured information that's easily accessible to those models. So looking at the problems we've faced and then the platform that we looked at building obviously fragmented clinical data uh developer friction and rigid infrastructure. These were historically issues that we were facing at data van as well as some of our uh other companies that we've acquired over the years.
So the inconsistent data formats have made it hard to really uh run models or analytics across that data easily. Developer friction it was painful for our developers to easily develop new products on top of our data. Uh that is something that is core to what we're trying to do at data van. We really want to unlock the developer uh workflow to allow them to more easily innovate. So what are our platform objectives? Obviously looking at unifying the clinical data processing.
Uh going back a few years, we had numerous different pipelines that we're processing data through. They were all separated again in silos, not talking to each other. So we wanted to unify that across the organization. As I talked about, we needed to accelerate the developer velocity. We really wanted to enable our developers and our product managers to be able to come up with new and innovative solutions to be built on top of our data. uh and obviously supporting the diversive workloads that we have here.
Our whole workflow really starts with the ingestion of data. So we're ingesting all of the different types of data that we have access to. We have huge amounts of PDF documents that we're receiving either provided directly to us by our customers or we're actually going out and acquiring them. Datavan itself has thousands of individuals who are out in the field actually embedded in hospitals and practices who are manually gathering charts for us only because we don't have those digital connections.
We're moving much more towards the digital integrations but still today I'd say the bulk of the charts we're collecting is still manually done. So anyway we're coming back we're collecting all of these charts we're doing conversions of that into sort of structured more standard patient centric data sets. We want to be able to create longitudinal patient records that give full insights into the individuals themselves.
Obviously sitting on top of data bricks, we do leverage the medallion architecture. That raw underlying uh layer is where really our immutable raw storage sits. Uh if ever we need to go back and look at that data, it's always sitting there. That silver layer is where we're doing standardization of it. We'll run PDFs through OCR. We'll pull out that textual content content etc. Uh we'll look at standardizing the vocabularies like ICDS and RX norms to make sure we're talking in the same language across the data sets.
And finally that product layer is that gold layer which is really driving a lot of the AI use cases. It's also driving a lot of the application use cases as well. Being in the healthcare obviously we were thinking about privacy uh from the beginning. We leverage uni unity catalog heavily for our role-based access control that allows us to provide the fine grain access either for our internal engineers and developers or our customers.
So we can provide that access to the data sets that they need without compromising our security. Data lineage and tracking is important. Uh we keep audit trails of everything from data ingestion, transformation, AI, everything all the way through the pipeline. So we're able to track any of the changes that we've made, any of the information that comes out at the very end to our end users, we're able to go back and see how that was generated.
So those audit trails for us are extremely important. Lastly, the time travel and change history. Again, this allows us to if ever we see issues, go back in time to see what the data looked like previously. So this is again critical for us to ensuring that we're meeting our compliance standards. Pulling all this information together, this really allows us to create this longitudinal patient view. When necessary, we're able to tokenize this data and deidentify it that allows us to securely uh merge or connect patient records that may have come from multiple sources in the back end without exposing that PHI.
This is important for our life science customers when they're doing uh clinical trials and research studies. So this is what come back to that you know trillion records that we're tokenizing annually. We do feature engineering. Uh we've got teams that are able to develop features such as lab trends and diagnosis combinations. This allows us to reuse that information across multiple use cases without having to redevelop that.
So what was this like before and what is this like now? Datavan over the years had developed their own sort of data platform. If I go back probably seven or eight years ago, we started looking into developing our own data lake. We also went down the whole road of developing our own ML platform. And over the years, we had developed a fairly robust ML platform that we were leveraging in house. It provided us very scalable infrastructure to be able to process huge amounts of data.
The problem with this was it was very operationally heavy. It was a big burden on our DevOps and our infrastructure teams having to maintain that stay on top of everything make sure we were patching and patch getting rid of any security vulnerabilities. So over time it just slowed the velocity of our development as we were so reliant on those teams to continually maintain it. Moving towards the data bricks uh platform has really enabled us to accelerate this a lot faster.
We've reduced the time greatly that it takes us to spin up new clusters for development or for uh model processing projects that we're doing for customers. The time that it takes for us to set up those projects is greatly reduced. And one of the best metrics that I think we have here is the 80% reduction in efforts for us to deploy new models into our data pipelines. We've baked in full CI/CD workflows in our model development life cycle, easily enabling our data science team to deploy new models and use cases as they see necessary.
So, as I talked about it, a large part of what we're looking as we think about an AI future is really wanting to provide structure to this unstructured data. We're ingesting that unstructured data. We're normalizing it through our processes and making sure that it's then available and stored within clear tabular formats. This allows us to develop those AI models, run those analytics against this data at scale. So unlocking that information within the unstructured data is critical and what we think is necessary to really be able to feed into the future of AI models that are going to come out to drive the best use cases and drive the most accuracy.
We've spent a lot of time developing our own in-house AI models as well. Over the years, I would say we've focused primarily on risk adjustment use cases. So we've developed AL models specifically for detecting the presence of conditions within unstructured data. Uh these models have been trained over time and fine-tuned. We continually evaluating them against the open market and seeing what other models are out there and we consistently outperform them.
At the same time, we continue to look at what are the new technologies, what are the latest in LLMs, seeing what use cases they can unlock. So looking at LLMs as potential use cases for can we convert unstructured data into fire by leveraging some of these technologies. The answer I think in a lot of cases right now is not yet, not accurately. But we're setting ourselves up so that when that time comes that that we think it's viable that we're in a position to be able to feed them in.
We can plug in the new models quickly and make take advantage of it. One of the cool things I think that we've developed over at data vent is our own in-house CQL engine. So CQL clinical quality language is a a language that can be used to query clinical data. It's primarily designed for looking at quality metrics a large p across large patient cohorts. We looked at some of the open- source technologies and the models out there.
Uh the problem that we were having was they just weren't performant for our use cases. We really wanted to be able to query large cohorts of data uh and see get results in very short time. What we did was we developed our own in-house CQL engine in Rust wrapped it in Python runs on top of data bricks. So it's able to run across extremely large cohorts of patients efficiently. uh the speed improvements that we get with our in-house model is magnitudes faster than what we see out there in the market today.
Looking at sort of a real world use case for our CQL engine, I said CQL was primarily designed for quality looking at quality metrics. We do see plenty of other use cases potentially out there for it for really just querying the data that we see. Trying to identify high-risisk patients within a patient cohort was one of the areas that some of our customers want us to look at. Some of this may be for looking at seeing what charts they want to chase for coding uh coding opportunities. uh and there are other use cases out there as well by combining both the insights that we get from the unstructured data by converting that into structured information as well as the structured data that we receiving whether it's lab results or claims data it allows us to enhance our CQL queries to generate much greater accuracy than we would if we weren't looking at that unstructured data.
So this is one of the real-time case studies for our CQL engine. We're still looking at other use cases for it. Uh and we're going to continue to develop this inhouse to improve it as well. So, so what's next for us as we continue to build our platform on data bricks? I think the CQL engine is one that we're going to continue to develop. As I said, right now it's extremely performant for us. We think we can improve that even again.
We want to be able to query across large cohorts of patients and get very short low latency response times for this. We're continuing to develop it. It's being rearchitected right now and we hope in the near future that's going to unlock some additional use cases for us. We are looking at future applications use cases whether it's automated provider coding uh whether it's clinical data registry building. So there are other use cases that we're looking to build on top of data bricks as well. some of what we'd like to be able to unlock.
Can we help identify clinical risk uh you know before those symptoms appear? Do we have that ability by looking across that data? We want to continually adapt workflows based on new patient data and outcomes as that's flowing in and as those patient longitudinal records are changing over time. We want to be able to deliver like personalized treatment recommendations at population scale. We want to be able to provide our customers the ability to do that based on top of the data that they're they're giving us and the insights that we're unlocking in that data.
We don't have to build all of this natively ourselves. We want to be able to provide our customers the access to do this with the data, but we're giving them the greater insights into that. So, we've got a lot we would like to do over the years. We're going to continue to build on top of the data bricks platform. Uh I think our relationship with data bricks has been strong. Uh we've been looking lately into some of the tools like DB Ignite uh and seeing where we can expand some of those ingestion pipelines.
So what are some of these key takeaways that we've had for healthcare organizations for the technology leaders and for data van ourselves? I said we've moved to this towards this modern architecture. We've been able to deprecate a lot of what we used to do in the past. That's reduced a lot of the operational complexities that we had with maintaining and managing our own platform. This has allowed the sort of development teams to increase their velocity.
It's allowed smaller teams to come out and build AI and ML models and plug them into various workflows uh improve either the either the workflows that we have right now or develop new use cases. The product team are thinking of new potential product opportunities now that they've got access to this data. the data science team is able to sort of do their own research studies and looking what information they can unlock now that they have the insights into that unstructured information.
I think for ourselves these architectural decisions you know said really compounding into those faster innovation cycles as smaller teams are delivering more to what uh to what we're able to deliver to our customers. That's all I have right now. I guess I wanted to open up and see whether there are any questions from anyone here.

Learn more about the Databricks Data and AI platform.