Unlocking Antibody Discovery at Scale: Vector Search with Databricks

Summary

Amgen uses Databricks Vector Search to semantically search 600 million antibody sequences by generating embeddings with the Amplify protein language model, treating antibody sequences as strings of tokens that can be compared by similarity rather than exact string matching.
Databricks AI Query with batch optimization enables efficient embedding generation at scale, and the resulting searchable vector index allows researchers to identify therapeutically relevant antibodies using statistical enrichment analysis across massive sequence libraries.
The lab-in-the-loop workflow combines computational discovery on Databricks with experimental validation in the lab, creating a feedback cycle designed to scale beyond 1 billion sequences as drug discovery programs grow.

Unlocking Antibody Discovery at Scale: Vector Search with Databricks

Antibodies are nature's most sophisticated molecules, and understanding their vast design space requires tools that scale. This talk demonstrates how to use Databricks Vector Search to efficiently search 600 million antibody sequences using embeddings from the Amplify protein language model, and how to build production pipelines that can grow beyond a billion sequences.
Learn how to embed massive datasets efficiently using Databricks AI Query with batch optimization, create searchable vector indexes, and perform statistical enrichment analysis to identify therapeutically relevant antibodies. Discover how a lab-in-the-loop approach combines computational discovery with experimental validation to accelerate drug development at scale.
🤝

Chapters

FAQs

Why use vector search for antibody drug discovery?

Antibody sequences can be represented as strings of amino acid tokens, and semantically similar sequences tend to have similar biological properties. Vector search using embeddings from a protein language model like Amplify finds antibodies that are functionally related to a query sequence, which is far more powerful than exact string matching for identifying viable drug candidates.

How does Databricks AI Query accelerate antibody embedding at scale?

Databricks AI Query allows batch model inference directly within SQL or DataFrame operations, eliminating the need to move data to a separate inference server. With batch optimization, it efficiently processes 600 million sequences through the Amplify protein language model, which would be prohibitively slow with single-record inference calls.

What is the lab-in-the-loop approach to antibody discovery?

Lab-in-the-loop means that computational predictions from vector search and enrichment analysis are sent to the laboratory for experimental validation, and the experimental results feed back into the computational model to improve future predictions. This cycle accelerates the identification of viable drug candidates by grounding computational findings in real biological data.

What is statistical enrichment analysis and how does it help identify therapeutic antibodies?

Statistical enrichment analysis compares the frequency of specific antibody features in a query-relevant cluster against the background sequence library. Features significantly over-represented in sequences similar to known therapeutic antibodies are flagged as potentially relevant, helping researchers prioritize which candidates to bring into experimental validation.

Full transcript

[00:07] Hello everyone. Just going to put up real quick um the forward-looking statement. Reminded everyone to complete their surveys. Going to start with a quick introduction. Hi, I'm Robert. I'm the director of therapeutic protein design at Amgen, a large
[00:23] pharmaceutical company. This is Hi, I'm Peter Hawkins. I'm a specialist solutions architect at Data Bricks. Yeah. We're going to be talking about unlocking antibody discovery and I'm going to give you a bit of a warning to start with. So, my background is
[00:38] evolutionary biology, physics, and machine learning and I have to spend the first 20 minutes of this talk mostly going into background information on the first one. We have a famous saying in biology, "Nothing makes sense in biology except in the light of evolution."
[00:54] So, for the talk to make sense, I'm going to have to give us some common language there. So, to start with, real simple, what is an antibody? Now, an antibody is a protein. A protein is the thing that your genes
[01:10] exist to code for. Uh it exists as a molecule in the body, which is a nanoscale machine. They can do a wide variety of things. They can block other proteins. They can pull things together. They can do actual mechanical enzymatic function.
[01:26] But, they are also just code. They are written out as a string of chemicals. You can represent them in a computer as an actual string. You can also represent them as tokens. That is a totally appropriate representation for what they
[01:41] are. Now, the molecule that they make, that code is basically op- operates uh as a processing unit through physics. Like, the code has a physical reason to do a certain thing. But, the author of
[01:57] the code is evolution. Um the compiler, the programming language it's written in, all of these are coded for the same way by molecules that evolved. Now, this antibody drug here works by binding a specific target. That's a
[02:13] really common property that they're capable of. Um we're mostly going to just talk about the sequences, the strings that lead to that. Now, antibody drugs are a very large part of medicine right now. This is one of the
[02:29] fastest growing uh categories of medicine. I think we passed a half billion dollars in revenue last year. Um what we're talking about here is a general category called biologics. These are called large molecule drugs. Now, when I say large, it is to compare them
[02:45] to traditional synthetic chemistry. Uh a drug by synthetic chemistry is usually about 50 atoms. These are about 100,000 atoms or more. They are a totally different scale of drug. And the reason that's important is cuz it
[03:00] actually gives you a lot more control over what they do. They really are nano machines. Uh an example of a class of drug that is part of this 500 uh billion right now are uh T cell engagers. Small molecules, they'll block little pockets. A T cell engager is a protein that will bind a
[03:17] cancer, and it will bind an immune cell, and it will bring the immune cell to the cancer to teach it how to kill it. Um the types of things you can do with proteins are significantly more complicated and significantly broader in
[03:33] scope than what you can do with a 50 atom small molecule. Now, there's an important note here, which is that almost everything we're doing in this space is coming from antibody derived molecules. There's a very important reason for that.
[03:49] The main failure mode of a large molecule drug is um an immune system reaction. The immune system evolved to detect and neutralize large molecules. It mostly can't see small molecules. But if you use a protein as a drug and the
[04:05] immune system picks up on it, the first thing that happens, and this is the best case, is that it will clear it from the body and neutralize it, which means your drug's not effective. Now, as anyone who has an allergy knows, the immune system's response to a perceived threat is not always graceful.
[04:22] You can have an overreaction to your drug that causes significant harmful effects, even death. There's a very simple rule in medicine it comes first, you can't do that.
[04:39] Now, to understand how to avoid the immune system, you have to know what the immune system is. And it's a complicated thing. In one particular example, how antibodies are made is very complicated. This is adaptive immunity. The immune system, like your body's a machine, it's a molecular machine.
[04:55] The immune system is a machine learning algorithm that does brute force stochastic optimization with like this genetic evolutionary loop. I'm going to go through each of these steps cuz they're all important. To start with, your DNA does not code for a single antibody.
[05:11] It doesn't even code for antibodies. It codes for a toolkit of pieces that an immune cell can put together to form a really wide diverse array of antibodies. Every immune cell then creates one antibody, and you have populations of
[05:28] just trillions of these cells. Now, these antibodies are random, they're naive. Some of them would bind to yourself, and if that happened, the immune cell would be used to attack yourself, an autoimmune disease. So, the first thing that happens is a screening step where you get rid of the immune
[05:44] reactive antibodies, the ones that cause problems in the body. It's a neat process. That cell, if it binds something it finds in the body, the first thing it does is it destroys the antibody it just made the gene, and then it chooses to kill itself
[06:00] gracefully in a way that releases itself back as nutrients to the body. After that, you got a population of random antibodies. When they're put out into, you know, the random tissues, anything that they bind they now know because they didn't bind
[06:15] it before is foreign. And if they bind something, which is a foreign pathogen, they go through a process of first amplifying. This means that they duplicate themselves. Every 8 hours you double the population. After about 7
[06:31] days you have billions of them. It's an amount of cellular tissue like the size of half a grain of quinoa. And that population of cells then attack the pathogen and clear it. This is why the common cold takes 7 days to clear. That process of amplification takes 7
[06:48] days. Those antibodies then, you have this giant population of successful antibodies, something that actually has a function. They then go through a long-term process of mutation and continued selection to gradually improve over time. That's the root of long-term
[07:05] immunity. Now, over your lifetime, this system will create roughly 4 quadrillion unique antibody sequences. That's uh exabytes of data is what that is. And uh
[07:21] as a machine learning system, when you know you're doing your vibe coding, you've got an agent running a bunch of tokens, just remember like however long that it takes for you to drink your coffee, your computer is burning through significantly less tokens than your immune system does just every every minute of every day.
[07:37] This 4 quadrillion is different for every person on the planet. It is the largest machine learning system that we know about. Little quick thing just on some terminology so that people know. The
[07:54] parts of your DNA on the top, your DNA codes for like a toolkit, there's certain pieces that we have to talk about. There's the VDJC. These are different segments that can get swapped in together Lego-like. At the bottom, the antibody sequence, you
[08:11] can find from the V, that's one part of the DNA. Like you have a whole bunch of V's you can swap together. The D is a whole bunch of those. We call them cassettes or segments. Your germline, your DNA from your parents has what's on the top. An immune
[08:28] cell has the DNA on the bottom where it has already combined these together into a functional way. So, when an antibody is created, it's not inherited from your parents. It is something an immune cell has created
[08:43] using your genetic repertoire. There's a really important detail about how it does this. Not all of that is from your DNA. The junction sites when it puts them together is actually random. Not completely random, biased. It is novel to the immune cell. Every antibody
[09:01] was created for the very first time by the one immune cell that created it. So, this is a lot of novelty, a lot of diversity. The system exists to create that diversity. I will also note that if you look at the sequence there, that most of that diversity is to the right.
[09:18] There's these regions we call the complement determining regions. Complement means the thing you're binding. Um the largest and most diverse one is the one CDR3. That's where all of this random DNA gets thrown in.
[09:39] Okay. And so, that leads to this big data thing. There's two reasons we use antibodies. One of them is natively they avoid the immune system. The immune system exists to allow antibodies to function. It has done the screening process on the antibodies and
[09:54] the antibodies that come out of the immune system are typically not reactive. Um the other reason we use antibodies is because it is very easy to get antibodies to bind a target you want them to bind them to.
[10:09] What you do is you immunize a donor with the target. Exactly the same way you immunize get immunized against COVID. Just an injection of the target you want it to bind to. The immune system recognizes this is foreign and creates antibodies to it.
[10:26] You then take your donor, you harvest B cells from their blood. You just harvest the B cells, you get billions of them. And then you put them through uh sequencing reactions. Now, sequencing DNA follows its own Moore's law. It gets exponentially cheaper every year. At
[10:43] this point it is completely reasonable to sequence a billion cells and get a billion unique sequences out of it. This gives you the repertoire of that donor. For a project where you're trying to make drugs, you might do this with 40
[11:01] donors. You might do a hundred of these projects a year. This is done at every large pharma company. It's also done publicly. Um and a significant amount of these repertoires get published and released. What we're going to be talking about here is a very specific data set that we
[11:18] can talk about because it's public, the observed antibody space. This is a collection of thousands of repertoires that are publicly available, curated by the Charlotte Dean lab, and available in a final set that when you look for uniqueness is about 600 million
[11:34] sequences, which is a perfect scale to talk about for Peter. All right, thank you, Robert. Yes, so we'll we'll take a a brief pause from the science, talk about some of the technical stuff, and and then go
[11:50] back to the science. So, I'm mostly going to be talking about how we used Mosaic AI tools to accelerate some of these efforts and sort of build pipelines to do this kind of work. So, if you think about if we take the observed antibody space as our example
[12:06] data set. You know, firstly, we have to ingest that data into data bricks, put it into, you know, delta format tabular data so we can start doing sort of distributed processing, do some QC, you need to remove some bad sequences, that kind of thing. And then after that there's two key
[12:21] steps, right? One is we have to A key thing we're trying to do is understand for a new antibody that we're looking at now in a new repertoire, what other antibodies have we seen that are similar to this antibody? And Robert's going to talk a lot more about that towards the end of the talk.
[12:38] But to do that, we have to embed these antibodies with machine learning models. And so, the question is, how can we efficiently embed in the case of the observed antibody space, 600 million of these protein sequences? How can we do that efficiently and get their embeddings? But obviously in the longer term for this problem where there
[12:53] are so many antibody sequences, this has to be scalable beyond that 600 million. 600 million is already pretty big, but we have to go beyond that. And then at the end I'll talk about how we can then build a vector search over those embeddings to be able to do that quick find other similar antibodies.
[13:09] I'll spend most of the time when I'm talking about this about the embedding part. And and part of the reason for that is I think, you know, if you think about building an embedding pipeline from scratch, if you were to sit down and write from scratch code to do embedding of this large number of of potential inputs,
[13:25] that's that a, you know, a non-zero feat, right? If you want this to run any reasonable time frame, you need to use many GPUs, which means you need to stream data from your centralized place where you keep your data onto those GPUs. And you have to stream them off again.
[13:40] Unlike training where you're you're maybe updating weights, in the case of inference, right, you actually have to move the the embeddings you create have to leave the the GPUs and come back to some centralized store. And while you're doing both of those streaming tasks, you want to ensure that your GPUs are actually being used very efficiently during that period of time.
[13:57] So, those data IO streams shouldn't be interfering with the GPUs. And while you're doing all this, another thing you want is is fault tolerance. If you are trying to embed a billion sequences and you get to sequence 500 million and something happens to go wrong, you don't want to lose the 500 million
[14:12] sequences you already did. And you want to be able to pick up from where you left off easily. Um so, let's think about, you know, on Databricks, what does this look like? How how could you go about doing this? And I think there are three sort of main approaches that that one might take. Um the first on the left is
[14:28] to use user-defined functions with Spark. So, probably a a Spark probably a Pandas UDF function is probably what you would use. Um in this case, you would have to manage your own cluster of GPUs, um which, you know, it is very possible,
[14:43] but you you something you would have to do. Um one advantage of this, if you use Spark, is that you can take advantage of some of the things that come natively with Spark. So, for instance, Spark structured streaming streaming provides fault tolerance. So, you kind of get that out of the box with this type of
[14:58] approach. But, you do have to manage some things. Um you have to think about how Spark works, how is my partitioning going to work, how often am I going to load my model on those GPUs, how long does it take to load the model? There's there's some considerations. And so, today, I'll mostly focus on the the other two. Um
[15:14] not to say that the first wouldn't work perfectly well, but the other two I think are easier, especially at this scale. Um so, the one in the middle is serverless GPUs. That's what we we're calling AI runtime. I think it was announced yesterday in the in the keynote. Um
[15:29] so, this gives you basically full control. You can spin up a whole bunch of GPUs. It spins up really quickly. And you can write your own torch data loaders. You can define exactly what happens as the data streaming in. You can define your own way that data streams out. So, you can basically do whatever you want.
[15:46] The caveat is that you have to then decide how you want to do whatever it is that you want. So, there's some overhead in terms of developer, um you know, maintenance of quite a bit of code if you do that. But, it might allow you to eke out extra performance when you have something where you think there are some tricks you could do where you could really eke out that extra performance.
[16:03] And then on on the right-hand side of AI query. So, this is a SQL function in Databricks. Um so, it allows you to basically call uh models served on Databricks model serving very efficiently. And because it's a SQL function, that also means that you can once again
[16:19] take advantage of Spark. So, you can once again take advantage of some of the things that come natively with Spark, which are which are very convenient. So, I'm going to talk a little bit more about each of these. So, firstly on the serverless GPUs, I'll I'll talk a little bit more about that, and then we'll go into AI query, which is what we used for
[16:35] this project. And I'll go through how we did that and and some of the considerations to make that as efficient as possible. So, with AI runtime using these serverless GPUs that spin up very quickly. So, one thing I want to note is that there's a really fantastic developer experience with that. Um
[16:52] you can spin up H100s, A10s like really very quickly. And you can have multiple nodes of those. Um obviously, you will then get to decide exactly what you you run on them. Um and importantly, there is a CLI now as well to allow you to submit jobs from
[17:08] your local machine. So, I think a lot of like machine learning developers are used to being in the IDE developing stuff. You can submit directly onto these GPUs from your local machine. Um and it's very tightly coupled with MLflow, which means that you can track So, if you're doing training, for instance, right? You can track in
[17:23] MLflow, and you can see in a nice UI how everything's progressing. And you can also see how the GPUs are being utilized. You can do exactly the same thing actually for inference as well. So, you can even track things like what's my throughput rate on every nth batch. So, you can actually track things like how well things are progressing as well as um
[17:41] uh how the system itself is is behaving. And there is also a private preview now for Docker support um for for AI runtime as well. So, that's that. I think when you really want to get into the very nitty-gritty, and you think there are some very principal things you could do to eke out
[17:56] extra performance, AI runtime's going to be your friend. But, there are many cases where AI query will also be quite efficient. Um and it and that's what we ended up using it in this case with a few little tweaks to make it extra efficient, which I'll I'll get to
[18:11] in a couple slides time. The The basic idea of AI query is if I take a model, and I wrap it with MLflow, and I serve it on a model serving endpoint, I can choose to have that that model served on some scalable number of GPUs. It can scale to zero if
[18:27] I want, so I don't pay for it when it's not being used, or it can scale up to, you know, very large number of GPUs, which means I can get very efficient throughput. And so, if you think about what you have to do to make that happen, there's there's I mean, there's two examples on the right, right? One is for using a standard sort of LLM for sort of text
[18:43] generation. And the lower one is a more classic machine learning model, where you take a row of data, and you, you know, take those different columns, you put them as inputs, and and you get an output. If you imagine you have 32 kind of threads or processes on your model
[18:59] serving endpoint, if you send 40 40 rows individually, eight of them obviously can't run it, right? Because you have only have so many processes. If you wrote something like this yourself against a URL where you're hosting an endpoint. You have to consider if things fail because there
[19:15] wasn't enough processes at that time, how long should I wait before I retry? If that fails, how long should I wait next time? And And there's some over like, you know, overhead of how you write that kind of thing. And AI query is a very efficient way to do that, and it takes care of all of that for you with a nice um
[19:30] SQL function. And so, as I say, you can do that with LLMs, you can do it with um classic machine learning models. Um and and this ability to run multiple processes at once can be very efficient. But there is one other thing you can do to make things even more efficient, which is to
[19:45] say, I know that when I run a single row, a single entry, as a batch through a GPU, that's less efficient per row than if I send multiple rows at once through that GPU. And so, what if I combine these two approaches? And this basically happens
[20:00] out of the box for the kind of accelerated ones on Databricks. So, most of the standard LLMs, most of the standard embedding models, kind of do this anyway. Um and and and a very efficient. Um but if you if you have a classic machine learning model that's
[20:16] um you know, some very custom architecture, there's no way for Databricks, for instance, to know how many rows at once can I put through that GPU. If I put too many rows at once, I will blow that GPU up. So, it's a bad idea to just sort of guess and blow up your GPUs, right?
[20:32] So, what you can do is is basically a trick is to say, "Why don't I combine this batching process myself? If I write my model such that I can send lists of inputs and then get a multi-dimensional output, I can send
[20:48] sort of microbatches to my GPUs and have AI query take care of all of the retries for me. And that can give you an additional boost um in performance. And it's a pretty low lift to actually do this. Um to change your um
[21:08] the the the model itself to to to have instead of a single input, have like multiple inputs and take a multiple output, is probably like I don't know, two or three lines of Python code. It's really not very much. Um and then what do you have to change on the inference side? Like when you write your SQL code around AI query, what do you have to change because you
[21:24] you modified the the the the model in that small way. Not too much. Um you basically have to just say I instead of having single rows, I want to pre-group my rows. So you just say you have basically a small you know, a couple lines of SQL to say
[21:40] bunch my rows into arrays. So I'm just going to have an array of all of my antibody sequences and maybe I'll do 100 at a time or 50 at a time, depending on how much I think I can fit on the GPU. That does require some experimentation, obviously. Um but you know, basically one line of SQL to to group into batches.
[21:57] Then AI query on those array types that get sent to the endpoint and back. And then at the end you just explode the results back out and you end up with a table where every row is one of your antibodies. And you have the antibody embedding with every row. Now that whole process is still wrapped
[22:13] as a single SQL expression. And this because it's just a single SQL expression, now I can still wrap that with Spark. I can take advantage of fault tolerance with, you know, what's that? Like one one or two lines of Python depending on how you want to define the object-orientedness of the of the
[22:28] dot. Um and so yeah, by doing this, you can still make really efficient pipelines. With the ability to take advantage of these Spark um functionalities. And then once you've done that, right? You've built this table where you have
[22:45] antibodies and their embeddings on every single row, you can now say, well, how do I build vector search over that? Um I'm sure many people in this room have have tried Databricks vector search for different problems. Even as we get to this very large scale where we're talking about 600 million antibodies or, you know, for you know,
[23:01] internal data sets at many companies that would probably go into the billions. Um, that's not an issue. You can still do the nice thing where you say, "I have a Delta table, and I want to make a vector search index." And you click a couple buttons in the UI, and you can do that, or you can write a few lines of Python
[23:16] and do that. We have the Delta sync, so that if you had this as part of a pipeline, like maybe you're sequencing antibody repertoires every day or every week, and you're updating your Delta table with new embeddings as they land, you can have the vector search index stay fresh
[23:32] as that new data lands, too. And Databricks has two, um, types of vector search endpoint, a standard one and a storage optimized. For this scale, this is what we use. I think anything above about 10 million, you you're better off using this one. So, it builds a lot faster, which is obviously
[23:48] important, um, and it can support up to this large number of of of embeddings. And so, that's what we use here. The latency is a little slow, but it's still sub-second. And if you push above a billion, you can always build multiple endpoints, and then just asynchronously search them. In that way, you can get
[24:05] to, you know, the ability to search beyond a billion antibodies. Um, and so, with that, I'll pass back to Robert, who'll tell you some of the cool things they've been able to to do with this. Okay. So, now we're into the title of the talk.
[24:20] So, I'm going to start with a visualization of the data. Um, this is a UMAP projection of the embeddings that we are using for the vector search. Uh, I'm going to mention very quickly that the model that we're using for this is a model that we uh
[24:36] train and open source with a machine learning consortium uh in Montreal called Mila. Uh, the model's called Amplify. Uh, this is a model that has trained on antibodies, but very specifically has not trained on any of these antibodies. This is an out-of-distribution set, and
[24:52] this is effectively a zero-shot representation. Now, it's important to note that every point in this data set is code. It's coding for something that is functional, and it is tested code. These are sequences that
[25:08] have passed through the clearance program of the body and are not immunoreactive. They are candidates to be made into drugs. Now, um we do two things when we look at this space that are both important. One of
[25:24] them is, if you're using biology to make your drugs, they are the needle in the haystack that you're trying to pick out. Now, if you're training models to understand this space in order to design drugs, now this is the actual landscape that model needs to understand. And so,
[25:40] the models exist to understand this landscape. You can see that the current representation uh does cluster proteins by some things that we know very well about them. On uh the left here, uh the different types of antibody chains, cuz antibodies are actually made
[25:56] out of multiple proteins. Got the heavy chain, two types of light chains. They cluster in this representation. Um if you look further in, in the heavy chain, there's different families of that V gene, which is the largest chunk of the the warhead. Those also cluster
[26:14] together. And you can see that geometry has been learned by the model. Uh on the right, you can see that some other information, the metadata you have, for example, the species that that gene came from, uh is also showing up in the uh the representation space. So, mouse has
[26:31] been separated from human. It's an interesting thing to note. If we get closer into specific properties, I'm going to show just a couple of examples here of like what similarity means according to this representation.
[26:47] So, the two left panels are just highlighting two very specific variants of the same V gene. Now, humans have diversity, so you can have a position in the DNA like this gene is different in you than in somebody else. You have two
[27:03] copies of each one, so you can have multiple versions of the same gene. This V gene has a variant that differs by a single residue. And you can see that the model definitely does recognize that these are different and overlapping,
[27:18] which is appropriate. Like that's that's just the truth about those sequences. Um when we get into more fine detail over on the right there, the J segment is the end of the antibody warhead. It's a much smaller piece. And you can see
[27:34] that within a cluster, it forms a substructure. Like there's additional information in there. Exploring this space is actually the goal. Unlike natural language, where you can look in a document and know what it means, you can read it and understand it. Nobody
[27:50] can read these. The point of exploring this space is to understand something. And so, when we're looking into like how can we quickly find similarity? How can we test this? This is an exploratory tool. This is discovery. We are trying to learn the code behind the function of
[28:07] these molecules. So, where does vector search come in? Here's an example of what you can actually just do with it very quickly. You take a sequence somewhere in the space. Under a second later, you have the top
[28:24] 10,000 hits to it. Now, if you're doing cold start, maybe you have to wait a minute the first time. Um finding the top 10,000 hits by traditional methods is possible. And I'll compare it to like doing a similarity search before and
[28:42] after Google. It's like the Dewey Decimal System totally exists, but people don't go to the library to look these things up. It is a very significant burden. Those questions just don't get asked for the most part. Once this is on, you can start asking questions very quickly and your
[28:58] exploration of the space can be iterative. It can also be automated. Here's an example of a quick search you can do. The nice thing about your vector database is that everything in it is also indexed to the metadata.
[29:13] Now, in our data sets that metadata will often be measurements and assays we've done on individual sequences, but in all cases one piece of metadata you will have is where the sequence came from. And these sequences usually come from donors. So, I'm going to show an example
[29:29] for OAS there. Um, if you do if you look at OAS, one problem with the data set is that one of the studies involved in its creation sequenced a lot deeper than everyone else. And it actually produces 75% of the data set.
[29:45] And it comes from three humans. It's kind of a problem with this public data set. If you take a sequence from one of those people and you do the similarity search, you can see that the top 10,000 hits are heavily biased towards the same person. That the signature of that
[30:02] person's DNA has been captured. Now, on the right I took an antibody that is a known SARS-CoV-2 neutralizing antibody. It was discovered through repertoire analysis where they took people who are immune to COVID and they looked through the repertoires to find
[30:18] antibodies that can neutralize the virus. And notably those repertoires are not in OAS. And when you search OAS with that sequence, what you see is that it's not a match to the human donors in there. It actually doesn't get high similarity to anything. And very quickly you can
[30:35] identify this as a limitation of the data set. This is an exploration that it can now be done in minutes, previously just wasn't being done at all. Now, our data sets have significantly more metadata, and we can kind of
[30:52] control what we're looking at when we do these searches. This type of enrichment analysis is very important to us because when we're talking about a donor, our donors are something that we have immunized with a target of interest. And if you're doing analysis of a single
[31:07] repertoire, you can try to find the binders to the thing that you immunized the animal with. But with a system like this, you can do that analysis across every repertoire you've ever done. And your statistical enrichment feature that you're looking for is if this if the things that are
[31:22] similar to this antibody are predominantly found only in animals that were immunized with the exact same target, you've now turned vector search into a score function for antibodies where the function is to bind that target.
[31:42] You also do get 10,000 documents. And again, these documents are code. If you want to look at that code and understand it, which is often our job, it's a very fast way to get the list. These are complicated pictures. I'll just describe them very quickly. Each one of these is a way to align a string and show the
[31:59] sequence probability at each position. The top will be the sequence that I put into vector search, and the bottom is the sequence distribution of everything that came out in the top 10,000. Now, you can see for the example from OAS that
[32:15] is mostly a match across everything. The V gene on the left is basically an exact match. And like I mentioned before, the hyper-diverse CDR3 on the end, that's where you start to see diversity. That's where you start to see this really interesting where does similarity start
[32:31] to break down, and what is a similar antibody according to your language model. The example I have on the bottom is the one that's out of distribution for OAS and you can see that it does not match the the very the Vgene framework at every position and there is very
[32:48] low similarity in the CDR3. That's an important thing to note when you have an out of distribution. This data set, while big, is nowhere close to the scale required to actually sample antibody space. Now,
[33:04] this type of analysis gets complicated, but it doesn't have to be complicated, right? Maybe you want to spend a long time looking at these 10,000 sequences and building machine learning algorithms off of them and all the rest, but there's actually something really simple you can do with a top 10,000 hit.
[33:20] Because these are primarily just diverse in a very small region, ordering a test tube of DNA that codes for all 10,000 costs $200 and can be done with an API. You can
[33:38] just fire off a request for that tube of DNA to show up in about a week at your lab. The assays we do with that DNA involve population analysis. You put those 10,000 genes into a bunch of cells and you can test those cells for what
[33:55] they do. Like, does the antibody bind to something? You can get a very quantitative measurement of the binding capacity of every one of the 10,000 sequences at scale by using systems that are almost entirely robotic at this point. Um this leads to a very simple
[34:13] lab in the loop process where you have a sequence of interest. You want to know more about it. You look in the data, there's no metadata for it. You don't know anything about it. Instead of just testing that one sequence, you can test the 10,000 closest
[34:28] sequences to it and then put all of that data into that database for next time. And as the models evolve, you can use the model to ask the question, which sequence do we need to know more about? And start populating your table,
[34:45] populating that landscape with real data that you explore in order to understand what everything actually is. So, I'll end with just a real simple conclusion slide. Now,
[35:02] typical drug discovery workflows, we are routinely generating hundreds of millions of unique sequences per project, doing projects every week. This is a totally routine pipeline that every company is doing. The scale that we're reaching
[35:17] is increasing exponentially. Costs of collecting this data go down every year. What we're actually keeping track of in data bricks is just shooting up. And having systems to explore it, we're talking about data that
[35:32] it's code that people have not been reading that we can now read quickly. Methods that people have been using that were bounded by, well, I can only look at one repertoire at a time, are now being opened up where you can do your analysis off of everything you've ever
[35:48] seen in less time than it took to do it the old-fashioned way on a single repertoire. It's definitely it's a transformative technology for where we're going, but it's also really the starting point. Cuz again, as I mentioned, your body makes exabytes of
[36:04] data, right? We're still just scratching the tip of the iceberg with it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.