When Data Saves Lives: Scaling Precision Medicine for Childhood Cancer

Summary

The ZERO precision medicine program at the Children's Cancer Institute manages more than 2 petabytes of genomic data from 2,800 patients and 14 billion variants, with one patient story illustrating how molecular diagnosis changed treatment from chemotherapy to immunotherapy and saved a child's life.
By migrating from fragmented text files to a unified Databricks lakehouse with Delta tables and denormalized schemas, the team can now query entire patient cohorts simultaneously using SQL, enabling researchers to identify patients with related variants across any genomic region.
Incremental variant annotation pipelines using Glow achieved a 42x speedup by eliminating redundant recomputation, and MLflow-tracked ensemble models with gradient boosted trees improved the Helium model's accuracy for variant prioritization in clinical decision-making.

When Data Saves Lives: Scaling Precision Medicine for Childhood Cancer

One childhood cancer diagnosis changed when precision medicine data revealed the true disease beneath the standard diagnosis, shifting treatment from chemotherapy to immunotherapy and a stem cell transplant. The ZERO precision medicine program at the Children's Cancer Institute manages more than 2 petabytes of genomic and clinical data from 2800 patients across 14 billion variants. This talk shows how data infrastructure fundamentally transforms cancer research and clinical workflows.
You'll learn how ZERO moved from fragmented text files and database silos to a unified Databricks lakehouse using Delta tables and Spark to represent complex genomic data. See how denormalized schemas with metadata columns enable querying entire patient cohorts simultaneously, how incremental annotation pipelines using Glow achieve 42x speedup by reducing redundant recomputation, and how MLFlow enables reproducible ML models for variant prioritization. Discover how building a feature store, ensemble methods with gradient boosted trees, and standardized evaluation metrics improved the Helium model's accuracy and clinical impact, ultimately helping researchers test novel hypotheses and identify patients with related variants across any genomic region.
🤝

Chapters

FAQs

How did precision medicine change a childhood cancer patient's outcome in this video?

This video shares the story of Caris, who was diagnosed at age 6 with standard leukemia, but ZERO's genomic analysis revealed she actually had Philadelphia-like ALL, a rare aggressive subtype. This discovery changed her treatment from chemotherapy to immunotherapy and a stem cell transplant, ultimately enabling her recovery.

How does the Children's Cancer Institute use Databricks for genomic data at scale?

The ZERO program migrated from fragmented text files and siloed databases to a unified Databricks lakehouse using Delta tables and Spark. Denormalized schemas with metadata columns allow researchers to query all 2,800 patients and 14 billion variants simultaneously using standard SQL, rather than processing each patient's data in isolation.

What is the Helium model and what does it do?

Helium is a machine learning model built to prioritize genomic variants for clinical review. Using ensemble methods including gradient boosted trees, tracked and versioned with MLflow, Helium helps clinicians focus on the variants most likely to be clinically significant, improving both speed and accuracy in precision medicine workflows.

How did incremental pipelines achieve a 42x speedup for variant annotation?

The previous batch annotation pipeline reprocessed all variants from scratch whenever new data arrived or annotation databases were updated. By using incremental processing with Spark and Glow, only newly ingested or changed variants are re-annotated, eliminating redundant recomputation and reducing processing time by 42x.

Full transcript

[00:09] Hi everyone, welcome to when data saves lives scaling precision medicine for childhood cancer. Um just some forward-looking statements. Uh don't forget to complete your surveys by the end of this be greatly appreciated and let's get into it. So when data saves lives
[00:26] at the children's cancer institute we believe a life should be long. So I want to start this presentation by sharing with you one of our patient stories and with the permission of Caris's parents I'd like to share with you her story. At the start of 2020 Caris was only 6 years old when she was
[00:42] diagnosed with leukemia specifically prebal one of the more common types of leukemia. When she was first diagnosed, her white cell count was so high, her oncologist warned her family that she might not make the next 48 hours. Caris, being the trooper she is, she made it
[00:57] through. But it was the first step in a very long journey. She was then treated with a standard regiment of chemotherapy, which made her very sick. But ultimately, Caris responded really well. By Christmas that same year, she had finished treatment. She was back at school. Life felt
[01:14] normal. monthly checkups, daily medication, but otherwise a regular life. Unfortunately, in Halloween, two years later, routine blood tests flagged something. Caris had relapsed and by that weekend, she was back at hospital.
[01:30] At the time, Zero, a precision medicine trial, had just expanded to all children with cancer. Before, Caris wasn't eligible, but now she was, and her family signed her up immediately. Zuro's genomic analysis completely changed the picture. She didn't just have standard
[01:46] leukemia. She had Philadelphia like ALL, a rare aggressive subtype. Her treating clinician said they would have treated her differently from the start if they had known. Without Zero and with her particular subtype of cancer, Caris would have followed a cycle of respond, relapse,
[02:02] respond, relapse until she wouldn't have made it. This diagnosis changed her treatment plan to imunotherapy and a stem cell transplant. And with that, Caris made it through. And last year, she was year six school captain. What
[02:18] changed her outcome here wasn't luck, it was data. And to understand this data, I'm going to give you guys a quick biology lesson. In you and me, there are trillions of cells. Every cell in your body carries the same instruction manual, your
[02:34] genome, your DNA. It's an alphabet of just four letters, A, T, C, and G. And it's around 3 billion letters long. Around 20,000 sections of this manual are what we call genes. Each one a set of instructions for building a specific protein. These are the molecules that do
[02:49] the work of running the cell. And all it takes is a single typo to cause a cancer. Let's go through an example. Let's take TP53. It codes for a protein that tells the cells when to stop growing. It's essentially the brakes. You change one letter and you build a faulty protein. The breaks fail. The
[03:06] cells keep growing when they shouldn't. This is cancer. And what we do is that we can record every typo we find in a text file called a variant core file or a VCF. Each row marks where the typo is, what has changed, and that's the kind of raw data we're working with. But a
[03:24] single typo is only the simplest way a cancer can start. To actually find what's driving it, we need more than one layer of data as that's rarely enough. We need several. So beyond single typos, we have four more layers of molecular data. Each one is just more text. So first is copy
[03:42] number and sticking to that manual analogy. Instead of changing a leather, whole pages get duplicated or torn out. A growth gene can be amplified many times over or a break gene deleted entirely. And to complicate things, what counts as a growth gene or as a break
[03:58] can be context and tissue dependent. The same gene can play a different role in different cancers. Second, we have our structural variants. This is where whole sections of the pip of the book get moved or flipped rather than a single letter changed. So two separate genes can fuse into one another and that
[04:13] fusion can switch on a cancer-causing gene that should have stayed off. Then we have RNA expression. This is how loudly each instruction is actually being read. Sometimes a broken gene is expressed far too loudly. Other times a gene that should be active goes silent.
[04:30] Fourth, we have our methylation which is the chemical tags that sit on top of DNA and switch genes on and off. You can think of this as glue sticking pages of the manual together such that instructions i.e. the genes cannot be read. And this is what drives cancer without changing a single letter in the
[04:46] sequence. Sometimes a single layer is a smoking gun enough on its own. But you can't rely that on every cancer across every tape, every case at every stage of a patient's journey. The only thing that holds up is reading all delays together.
[05:02] And that's something a targeted panel or a single test simply can't do. And this is why curing cancer is so hard. You're not trying to cure one disease, but thousands, all with their own unique signatures. These circus plots represent the molecular profile of
[05:18] a patient's tumor. And even of the same cancer, uh, they can look completely different. Different mutations, different structural changes, and ultimately different biology. And this is why standard treatment doesn't work for every child. The same disease name can hide fundamentally different cancers
[05:34] underneath. And precision medicine is all about using data, genomic and clinical, to tell them apart and treat them accordingly. And this is also why full data access matters. We need the full molecular profiling. Otherwise, we're only ever seeing part of the picture. that data that changed Caris's
[05:51] diagnosis was always there. It just wasn't accessible at the right time. So I want to quickly show you how to read one of these because I think once you can the point kind of lands on its own and you can read it from the outside in. The outer ring is the genome. All the chromosomes laid out in a circle. Uh
[06:07] we didn't have the next ring which is that typos that I was talking about before. These are called single nucleide variants or snibs. Um each dot each dot is a variance colored by what kind of change it is. Then we have copy number. This is where whole regions can be gained or lost the duplicative duplicated pages from
[06:23] before. And ultimately we have the lines crossing through the center and these are the structural variance. These are when genes are fused together or the rearrangement of the genome. So every one every one of these rings is a molecular layer we just talked about. And the whole point is that a single
[06:39] patient's complete profile can be represented in I guess a single picture. And this is kind of what seeing everything at once kind of looks like. Um, and this is why no two cancers are the same. So everything we just saw, the circus plots, the no two cancers are alike. Using that full molecular picture
[06:55] to treat cancer is called precision medicine. And I think it's easiest to define by contrast. So standard of care treats based on the average response across a patient group. What has worked best historically for this type of cancer? Precision medicine treats based on the specific molecular alterations
[07:11] driving this tumor. And to be fair to standard of care, this isn't old is bad, new is good. Standard of care tries to stratify patients already. For example, in childhood alll that common leukemia type that caris was previously diagnosed with uh establishes tests to sort
[07:26] patients into different risk groups. Caris was um in different risk groups and it works right. Uh cure rates for that kind of che that kind of disease is around 90% up from near fatal from almost 50 years ago. And this is one of the great medical history success stories. The limit of standard of care
[07:43] is how much biology it sees and how it treats it. The test may be targeted. They look for what they already know what to look for. And the therapy is largely systemic chemo uh killing all the rapidly dividing cells indiscriminately. And this is why it's so toxic because all the healthy cells
[07:58] are also targeted alongside the cancer. Precision medicine adds that full molecular profile to find the exact driver. then matches the therapy that blocks that specific pathway, sparing more healthy tissue, but also targeting the direct cause of that cancer. And
[08:14] that's exactly what happened to Caris. Standard testing diagnosed her leukemia. But it was the comprehensive sequencing that revealed a rarer, more aggressive subtype underneath, which changed her treatment and her outcome. She's at risk of being in that 10%, but the data moved her out of there.
[08:32] So, does this actually work at a population scale? I think the published evidence from our studies and netmed and netcom says yes and it makes a very specific point that matters for a data audience across a cohort of 384 high-risk uh patients giving
[08:48] comprehensive molecular profiling. The benefit wasn't a more fancier drug. It was choosing the right treatment at the right time guided by the data. Precision guided therapy beat standard care on a 2-year progression-free survival, 26%
[09:05] versus 12%. And when it was given early, before the cancer had progress, that gap widened to 42% versus 12%. But here's the most striking result. The data also tells you when it works best. And so free factors predicted the most benefit
[09:21] whether there is strong clinical evidence for the target if the driver mutation was a structural variant and giving the therapy early. If a patient had all threeear progression free survival jumped to 88%. And it holds even in the hardest diseases. The fuse midline glyomomas is
[09:38] a rare inoperable tumor in the core of the brain. It spreads through those structures. You can think of black ink on a sponge. So you just can't cut it out. It's near universally fatal. Yet after pro progression, precision guided
[09:53] therapy gave a median survival of 15.3 months versus 4.3. Three of those children benefited from targets that only became visible when we combined all the molecular data types. No single source would have surfaced
[10:08] them. Completeness of the data was the difference for them. The benefit comes from integrating every data type. molecular data to find the driver plus the clinical evidence and drug availability to act on it. So I'm James Bradley. I'm a bionatic
[10:25] data engineer at the Children's Cancer Institute and Zero is the precision medicine program I work on. It's a national clinical trial open to all children with cancer in Australia and is now expanding to include adults with pediatric type cancers. This diagram shows the zero workflow. A child is
[10:41] diagnosed, a tumor sample sent to us. We do in-depth genomic analysis, whole genome sequencing, RNA methylation profiling and the results get curated by a team of expert scientists and clinicians and ultimately shared in front of a multi-disciplinary tumor board who reviews the results with the
[10:57] treating clinician and recommends a personalized treatment based on the molecular profile. The program now covers 11 sites across Australia and New Zealand with roughly 13,000 patients per year. And for high-risk patients, those with less than a 30% chance of survival,
[11:12] Zero has doubled their survival rates. The goal is to make this standard of care and not just a research extra. And as you might expect, there are lots of challenges to overcome. And one of them is the data architectural challenge to organize all this information. For a
[11:28] large part of zero, we'd have the standard data lake and warehouse approach. Molecular data would be uploaded to our on-remise data lakeink trigger some pipelines which would run our bionatic analysis. This would then be loaded into our data warehouse which would serve our web variant curation platform zero dash. This is where our
[11:45] scientists and clinicians would go through the variants to identify driver mutations. This organization of information in our web platform Zerash took our curation time down to just 48 hours. But now due to the sheer size and complexity of genomic data, we couldn't
[12:00] load everything into our database. So we made a deliberate decision to only hold a subset of gene regions already known to be important in pediatric type cancers. Other molecular data was saved, but everything else was archived. And there's an irony there. We started
[12:16] this presentation talking about not seeing the full picture. And now we're bottlenecked by our own database, facing the same problem. We had fast answers to questions we had already decided to ask. But for research, we need a system that lets you ask any question and the
[12:32] freedom to explore that data on your own terms. Simply put, our database representation just wasn't enough. So we pivoted to a lakehouse solution to complement what we already had. Transferring data we want searchable into an ingestion databicks workspace
[12:49] and representing it like that VCF file from earlier as a delta table. Previously, every new research question meant writing a new script to search through a data lake. This is brittle, hard to reuse, and difficult to scale. The foundation of our solution is a raw
[13:05] delta table representing our data lakeink files where metadata like file name is structured as rowle columns. We then use Spark 4.0's variant type, think optimized JSON object, not to be confused with genomic variants, uh to
[13:21] store the file data itself. And this gives us consistent structure up front so that querying the data lake becomes like querying a highly optimized table with Spark as the compute engine. And this flexible format also solves integration since all of our data lives in one place. Combining diverse sources
[13:38] becomes use case driven. Once again, Spark handling the computations that SQL simply couldn't. And not even in the sense of scale, but in terms of the flexibility of the programs you could write. Finally, Unity catalog provides that scalable governance layer across delta tables, object storage, and ML
[13:54] models, giving both researchers and engineers proper access control at scale. And I'm going to run you through some use cases to showcase this. So, we're going to go through three use cases, and through each one of these use cases, I'll go through the motivation of why we're doing it, the architecture, the solution to solve the problems that
[14:10] came up, and ultimately the results. So, let's begin with our first one, whole cohort queries. So every tumor we sequence goes through Enkco Analyzer. This is an open- source pipeline that runs 25 different specialized apps, each characterizing
[14:27] different molecular dimensions of the cancer. Structural variance, copy number, RNA expression, mutational signatures, immune profiling, and much more. We get about 19 apps that emit a final output. some text files, some plots, and they all land in a strict
[14:42] folder hierarchy with analysis set ID followed by the version of Enco Analyzer, then the app name, then ultimately the files. An analysis set isn't just one sample. It's all the bio samples for a patient, tumor, DNA, germline DNA, and even the clinical for
[14:59] each clinical event for the patient. So, this data can really fan out fast. The point is this is incredibly rich, but it's hundreds of files of patients scattered across a directory tree in a data lake, which is not as fast as a directory tree in your file system.
[15:16] And here's the problem. Only a subset of these outputs ever make it to our warehouse to be served on Zero Dash. And we get around 300 million variants every month, building to our growing data set of around 16 billion variants. So if you're a researcher wanting to ask a question across the cohort, you're
[15:32] downloading hundreds to thousands of file files and searching them with a script. And I think this is the modern day equivalent of searching by hand. Every question is a new one-off script. And this is brittle, slow, and it just doesn't scale. What we actually want is to query this data across the whole
[15:47] cohort and join it with clinical context. And that is our use case. I want to start off the solution by first describing our data model and how we link samples together because I think it's the key on how we join everything together. No pun intended. The core
[16:04] entity is an analysis set and you can think of it as one clinical event in a patient's journey, a diagnosis, a relapse and it belongs just to one patient. A patient can have several analysis sets over time. As her disease progresses, we sample again at each event which lets us watch how the tumor
[16:21] evolves and what has changed. Each analysis set holds multiple bio samples the actual materials that are taken from the patient and these come in different tumor types tumor DNA dumi DNA etc. the same molecular layer molecular layers we were talking about before. Um and the
[16:36] shape is simple. Then you have patients which belong to they can have multiple analysis sets or an event and then they can have many bio samples and this directly translates to our output path uh layout in encoer and is
[16:52] used as the base for how our raw files are stored in our data lake. So our raw delta tables is a denormalized join between the data lake file paths and their contents. In this case files containing every mutation
[17:08] found in a sample. For example, we put VCF files that relate to the semantic which means tumor samples into one delta table which can have several programs or outputs or VCF bundled into one table. And to dive a bit deeper, the table's denormalized join is actually between
[17:24] the file standardized metadata. So the sample ID, the app name, the version and these are proper columns uh with the files contents alongside it. Thus allows us to have every row carry its own provenence. So you can filter by app version and a certain and be certain
[17:40] that the contents came from those exact files and the payload itself goes into the spark variant column once again optimized JSON object. uh so we get structure where we want it and the flexibility where the file formats differ and we use the lakehouse
[17:55] federation to then to connect to our SQL server which hosts our zero dash web app so the clinical context that is being described in the case is immediately available to be joined with these delta tables and so a very common pattern that we do is associating cancer diagnosises with varants
[18:12] so this lets us explore what mutations drive specific cancers and what you're seeing here are just simple filters and aggregations and even that alone we can see that the K28 mutation is heavily over represented in diffuse midlineomas compared to other similar central nervous system cancers and if this is
[18:29] what we can surface with just basic queries imagine what becomes possible with machine learning and it's exactly where we're heading data bricks ties that next step together with ML flow giving us a traceful lineage all the way from our raw data to our model and I'll show you showcase that to you in our final use case
[18:46] this allows us to have all 16 billion and counting variance It's queryable and joinable with real time clinical context. But it's more than that. Our data is now represented in a format that can actually scale. Onto our next case.
[19:03] So once a sample passes through all those bionatic pipelines, the variants are essentially frozen. You don't touch them again unless you do some sort of major pipeline version update. Uh, one of the big ones is when we migrated from the human reference genome HG37 to HD38,
[19:18] which meant reprocessing everything. But here's the thing, the variant itself doesn't change. What changes is what we know about it. Is it pathogenic? Is it in a tumor causing gene? Uh, is it rare in the population? And this kind of
[19:33] knowledge lives in these large public data sets, uh, Nomad AD, ClinVar, Cosmic, and they re-release roughly maybe every 3 months or so. So to give clinicians and researchers the most current picture, we need to reanotate our variance every time their sources
[19:50] update. And this is an example of what a curator or researcher would see in our on our web application. And so highlighted some of the fields where these annotations come from. And you can see in our free text section, our curators would reference a lot of these external databases to uh provide the context of
[20:06] this variant. And what I want to show you here is how we use the lakehouse to keep pace with that exponentially growing public data set. So many of these external data sets are just large text files that you can download and then you can represent them
[20:22] as a delta table and then join them against a raw variant table on some sort of mutation ID key prompt pro alt right and this is highly performant with photon once you have it all represented that way and it avoids red and we can then avoid redundant recomputation
[20:39] through incremental processing using spark structured streaming. From there, we can apply some of our domain expertise to optimize further. We know around 96% of incoming variants that we collect, we've probably seen before. So for our most expensive
[20:55] annotation step, VEP, which stands for variant effect predictor, which due to technical reasons cannot just be represented as a single table and is a command line program. We run that on that remaining 4%. And we do this using Glow. Uh this is a framework which can
[21:12] parallelize bash scripts across a cluster making it quite straightforward to plug in new tools and apply them at scale. And using Spark's unified batch and streaming architecture, we can control the flow of batch data. We use something like max files per trigger. We can thle
[21:28] how much data actually goes through our most expensive step at once. But this is critical running when you run a bash script or maybe some next flow pipelines. We know things can crash without microbatching. A single
[21:44] malformed row could bring down a 20our job and we have to reprocess everything from scratch. By breaking the work into small checkpointed chunks, a failure only ever cost us the current batch. It also allows us to keep us within our compute quotas. And since we're never
[21:59] pulling in more than a controlled amount at once, which you can optimize the wazoo out of. And so then incremental processing gives us that resilience and cost control in one. The result is a 42x speed up over our
[22:14] previous per sample VCF annotation pipeline largely because we can process all our variants together and remove all that redundant recomputation. And I just want to provide a quick word and one of like where that 42 comes from. So I took the average throughput, the snibs
[22:30] annotated per second from our old annotation job, the bash pipelines and compared it against this pipeline. Uh I assumed around a 20x parallelism for the old one. Um that's running the old pipeline with 20 jobs in parallel which is roughly the best we could manage with
[22:45] our resources. And the result is about a 42x speed up. But if we were to compare annotating each bio sample sequentially with the old method, it's closer to like a 900x speed up. But the bigger win might be the developer experience. The old method reantated huge number of
[23:02] duplicated variants and updating our core databases with that data through our loading scripts would take an infeasible amount of development time to get right that I can't even imagine it. So thanks to the glow framework which is I think was presented at this conference maybe back in 2019 and has been
[23:19] supported by data bricks employees in the past, we can simply swap out vep for even a faster version. And so there's this open source Rust implementation of EP which come out which boasts like 130x speed up which we're able to directly pipe into this pipeline. Uh and since we
[23:36] can package it all up into our docking container uh we can plug it directly back into the pipeline without changing the architecture. We're still really experimenting with this since fastf's output is slightly different than ve for a bunch of other binatical bionatic technical reasons. The main point is
[23:53] that we can benefit from improvements of research programs directly to our large variant delta tables. And this is a big thing in research, right? Because a lot of these tools published in the literature um as some sort of command tool. So by using data bricks and glow, we could rapidly try out these new tools
[24:10] as they come out. Onto our final use case, which is helium. And this is an example of how we upgraded a key machine learning model we use for variant curation. So how important is this variant? So now
[24:25] we have our variants are they're kept up to date with these public databases. Uh and we want to run then we run into another problem. How do you actually interpret them? There can be thousands to millions of variants per bio sample. Each one carries dozens of annotations, pathogenicity scores, other ML outputs
[24:42] predicting the variants impact, uh the role of a gene it affects and so on. It's a bit like the explosion of information you get when you analyze a cancer with all these different molecular types except now we're seeing that same explosion down at the single variant. Um there's just a lot more
[24:57] information to comprehend. So what we really want to do is filter out the noise to give you uh to filter out the noise to give you a sense of the scale. Uh in total we have around 15 million sematic or tumor sniffs. Uh of those 820
[25:14] were deemed reportable. These are manually marked as being of interest and clinically relevant to the patient's case. And that's across more than 2,000 cases. The usual way these variants get curated is fairly manual. You filter down to a list of known relevant genes
[25:29] and then have curators or clinicians search for specific genes they know are linked to the patient's suspected diagnosis. So the questions is can we curate and identify variance of interest more systematically? What if we can simply score a variant on how reportable it is?
[25:47] Some sort of recommendation system that can learn to identify important variants. And we do exactly that. Here's a view of zero dash on our snips tab. And this is where curators mark those snips. These are typos from earlier uh are of
[26:02] clinical significance. On the left, we have what is called a helium score. This is a custom model that assigns a score based on a set of rules to predict if a variant is reportable by turning through all those annotations before. The idea is that the most important variants float to the top, hence the name helium.
[26:20] You can see on the right hand side of the variant uh if the variant has been deemed reportable before and reportability is a umbrella term to denote the variant is of significance and we break them break down reportability to whether it's targetable uh by a known drug or if it's diagnostic
[26:35] indicator and we keep track of how many times we've seen this being reported in other cases as well and have direct links back to those cases. Also mind you, these variants are pageionated to the user. So curators rarely go through all thousands or millions of variants line by line and
[26:51] they heavily rely on filters to sort through this data. A bit more on the original model. As I said, Helium is a rulesbased model, a set of predefined rules that assign a score when a variant matches one or more criteria. It produces a score between
[27:07] zero and one. The higher meaning more reportable. The rules are based on real curator experience and but with a lot of like genetic risk scoring how much weight you assign each rule um is somewhat subjective. It's hard to say how much one signal should matter
[27:23] relative to another across every single case. So we still rely on humans to review and understand the full context of a patient's case. So for this version of helium uh has helped in thousands of cases. But we face a growing problem as we scale our precision medicine from
[27:39] just high-risisk cancer patients to all cancer patients. So there's an engineering reality here. The model is incredibly slow at inference and to break down on million variant runs. It was all written in pandas. Some documentation but no baselines, no evalues to work with and no version data
[27:56] sets in which we could point uh for evaluating performance. To make matters worse, the data scientists who built this left five years ago and that makes it really h really hard for us engineers to fix these issues and scale. There's no rigorous framework in place to improve it. And we couldn't make a
[28:13] change and say with any confidence what we actually made anything better. So one of the first things we did was recreate a trustworthy baseline. We had all the original Helium scores in our MySQL database for each of these variants. So we could use this as a
[28:29] baseline score on how um the old version helium version 2 worked. And then we rewrote the model from pandas to pispark and removed a lot of inefficient code. The original database was definitely created when zero had a lot less patience several years ago and I would
[28:45] say is classic data scientist code. This gave us a huge speed up and I generally find at the scale of data that we deal with in zero recently if I can just represent my problem with spark and delta I can get some pretty significant performance increases just from that. Uh this particular example had a lot of
[29:01] inefficient queries and and stuff like that which I think helped dramatically once you ironed them out. But regardless delta and spark make the whole hor horizontal scaling thing really easy. So once we had our baseline set up and measured its concordance with the old helium, we can really start to improve
[29:17] from our baseline. So how do we make better measurable? We start off by building off our annotation data set from the last use case. We then generate a feature store where we take annotations from the annotated variance
[29:32] table and generate the specific features needed for the healing model. Um an example of a feature is is this variant ankco gene? We already have primary keys being the mutation ID and sample ID. And now we're just adding labels if they've been classified as reportable. And this
[29:49] comes from our SQL warehouse from Zero Dash, the web app. And then we follow a pretty standard MLFlow training pipeline. Essentially, you split the data um by like an 8020 split based on a number of bio samples. Uh but what's important is that we log all of this into a single MLFlow experience
[30:06] experiment. We then try several different model architectures to see which one works best. U and these are all trained on the same training and test set as individual runs in that experiment. We then evaluated them on the same eval suite as well and
[30:21] comparing to the baseline rules model eventually tagging the best model as champion and registering it to our schema. So let's dive a little deeper into the evals. We standardized a set of eval things like top k accuracy which measures if the reportable variant is in
[30:38] the top k variance that like in that uh curation page before we didn't have recall which measures what fraction of variants appear in the top k positions of reportable variants we didn't have normalized discounted cumulative gain which takes into account the position of
[30:53] the reportable variants uh because reportable variants should rank higher than the non-reportable variants. These atk metrics are important because it's how the curator sees data on that pageionated app uh snips tab. We don't want them loading all the data at once
[31:09] into our web application. And just FYI, the step uh in the graphs represent the K. And we can see from these graphs that these trends there tends to be a model that dominates. And the model that we found was our rules plus gradient boosted decision tree ensemble method
[31:25] which performed the best AC across all the re valves. And so finally we can answer the question if making a change to this model actually helps. So we can dive a little bit deeper into the performance of this model and compare it to that baseline rules model to begin with. This step is where we
[31:43] actually look at which variants does this model perform better in. We define some other eval such as pass rate. This is uh what which is like if the variance defined within the top top k variance at that top 10 variance at that bias sample with their score being greater than 0.7
[32:00] or score failures. This is when the model reports a reportable variant being below a score of 7. And we can see that we get around a 5.4% increase in the pass rate uh with our new model reducing score failures from 84 in the uh in the rulesbased model to just 49. Right? The
[32:17] net outcome is a faster, more accurate variant scoring system and data bricks allows us to demonstrate a better model all the way from that source data. So just to recap, we've gone through having all this large amount of data uh being queriable and then enriched with
[32:33] real-time clinical context. We then use Spark structure streaming in a ledger of distinct variants uh that we've processed. This can reduce the amount of variance we need to annotate with our slower binatic tools just 4%. uh we then use data with ML flow and a feature store to build and demonstrate better
[32:50] variant scoring models and that link directly back to the data sets being used. We then have utilized the entire kind of lakehouse experience to go from data all the way from from the raw data to learning and then improving an outcome.
[33:06] So I just want to say all this work for the helium stuff was studentled. Uh all the results shown today are from my banana group but both teams came up some very excellent implementations. I've supervised two groups of final year computer science undergraduates as a part of a 10-week project with around an
[33:21] hour of two supervision per week for each term. Um and this is my second or third data bricks project I've supervised and anecdotally I'm finding it takes these teams around five weeks to really grasp what data bricks is and the context of the problem. I always ask
[33:36] the students how they felt the difficulty of the project was to begin with and they always think it's one of the hardest projects on paper. Uh the students get to choose from a list of industry partners to work with. Partially I think because it's the biology is probably quite intimidating
[33:51] as well as I think the goals we're trying to achieve seem really daunting without a system in place. But with data bricks I can provide those guard rails and allow them to work within a system to achieve these lofty goals. And once they get it and internalize how data
[34:06] bricks works by using it, they can really fly and achieve so much more than I could have hoped for. Simply put, data bicks gives us incredible leverage. The flexibility to apply our expertise where it matters with the data infrastructure that provides that 10x leverage. And for what
[34:23] we're paying, I think the value is enormous. There is a learning curve, but that's where much of the leverage compounds. Much of what data bricks is built upon is open source, but it's the way it's been put all together in a unified data system which makes it so powerful. I couldn't have stitched those tools together myself and gotten
[34:39] anywhere near the same value. It lets us focus purely on our expertise and our data while benefiting from a platform maintained by some world-class engineers. And honestly, I think I've gotten far more leverage out of data bricks than any sort of AI tool. And I
[34:54] love AI, but you combine the two and they're greater than the sum of their parts. So some quick takeaways. Use data brick asset bundles and infrastructure as code as early as possible. Ideally once your PC is done. It really makes all that complicated cloud stuff so much
[35:11] easier. The UI is quite good. So dive in, experiment, have fun. It really does start to click once you start using it. If I have some time, I'll go over some really practical monorreo tips for storing your data brick asset bundles. So we have one repo which holds all of
[35:27] our dabs for our pipelines. And what I have found after developing all these pipelines is that I'm reusing a lot of custom code. Therefore I put in this Python package in the same repo. I've called it de functions which stands for data engineering functions. And the point is that this package can be shared
[35:43] across all our dabs. So code is not duplicated. So if you're working in some sort of development dab for a dev environment, I add the shared Python package as an artifact of that dev deployment. I then use relative paths and the monorrepo to
[35:58] my de functions library. This makes sure that changes dev changes to my shared packages are reflected immediately in the dev environment. And then for production, I specifically use a CI/CD pipeline to share the official release version of the library to a shared
[36:13] location on our production data bricks workspace. I find this pattern quite helpful and I really wish I knew this when I started. So what's next? Our data is only going to grow. Pabyte scale at co as our cohort expands and new sequencing types
[36:30] like liquid biopsies come online, allowing us to have genomic surveillance through simple blood tests. The cohort is growing too. 13,000 patients a year and we have a lot of international collaboration on the way. Zero started as a research program medicine works. It's now a national
[36:47] trial across 11 sites. The goal is standard of care. The default for every child diagnosed and not an extra step. Getting there involves many pieces, clinical, regulatory, and funding. But a major component is the data and systems
[37:02] challenge. And that's what this talk has been about today. There's a lot more we're exploring too. Genie Lakeb and our own insights projects for data sharing. All of this is aimed to make data accessible to researchers and clinicians without writing any code.
[37:21] So Caris is in year 7 today. She just started high school. She has an incredible thirst for knowledge. She's making plans with her friends. She's watching soccer. And she's adoring her dashound Omar. She's calm, thoughtful, and mature beyond her years. Her family count
[37:38] themselves incredibly lucky every day she's still with them. The systems that we're building here isn't for some sort of abstract data problem. They're for this. Making sure that the right data reaches the right clinician at the right time for every child and not just the
[37:54] lucky ones. And that is what solving this data problem looks like. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.