From PDFs to Insights: How to Build a Clinical Trials Knowledge Assistant
Summary
- Clinical PDF processing: Agent Bricks Knowledge Assistant automates auditing of clinical trial protocols against FDA/EMA standards, transforming weeks of manual PDF review into minutes of AI-powered processing.
- Multi-agent scaling: Integrates text-based Knowledge Assistant (for clinical PDFs) with Genie on structured clinical data, creating a multi-agent system that bridges unstructured regulatory text and structured trial evidence.
- Domain learning: Agent learns complex business logic (like distinguishing ongoing from completed trials) through simple feedback, enabling precise queries about clinical trial protocols, results, and regulatory compliance without manual SQL.
From PDFs to Insights: How to Build a Clinical Trials Knowledge Assistant

Stop drowning in unstructured PDFs and start generating actionable regulatory insights. In clinical development, auditing a single protocol against FDA or EMA standards can take weeks. This demo shows how Agent Bricks transforms a mountain of chaotic PDFs into a precision-tuned Knowledge Assistant that understands your domain in seconds.
Key Highlights:
- Unstructured Data Mastery: Process diverse clinical PDFs and regulatory guidelines with ease.
- Task-Specific Optimization: Watch the agent learn complex business logic (like "ongoing vs. completed" trials) through simple feedback.
- Regulatory Intelligence: Automatically audit protocols against regulatory standards.
- Multi-Agent Scaling: See how to integrate text-based assistants with structured data tools like Genie on the Databricks Lakehouse.
Transform weeks of manual work into minutes of automated insight.
CHAPTERS:
📚 Resources:
Learn more about Agent Bricks Knowledge Assistant: https://docs.databricks.com/aws/en/generative-ai/agent-bricks/knowledge-assistant
Chapters
0:00Challenges in Unstructured Clinical Data0:36Architecture: Agent Bricks Knowledge Assistant1:06Data Foundation: Protocols & Regulations2:26Setup: Building the Knowledge Assistant3:52Live Demo: Querying & Discovery8:03Wrap-Up & Next Steps
FAQs
What problem does it solve?
Auditing a protocol against FDA/EMA standards manually takes weeks. The Knowledge Assistant automates this by processing clinical PDFs and regulatory guidelines simultaneously in minutes.
What data sources are processed?
The assistant processes clinical trial protocols (PDFs with varied structures), trial results, metadata about trials, and regulatory documents. All registered in Unity Catalog for governance.
How does the agent learn domain logic?
The agent learns through feedback. Provide examples like distinguishing ongoing from completed trials, and the agent refines its understanding for precise queries.
How does multi-agent help?
Knowledge Assistant handles PDFs while Genie queries structured data. Together they enable regulatory scientists to ask complex questions combining requirements and evidence without SQL.
Full transcript
0:00 Challenges in Unstructured Clinical Data
In the pharmaceutical world, the distance between a clinical trial protocols and a life-saving results is paved with thousands of unstructured PDFs. Right now, it takes teams of expert weeks to cross reference a single protocol against shift in FDA or MEA regulations. But what if you could audit the entire life cycle in minutes? Today I'm going to show you how Asian brakes turns a mountain of Q clinical data into a precision tuned knowledge assistant that doesn't just read your data, it understands your domain.
Let's dive deeper. So how does it work? We're going to select a few data sets containing clinical trial protocols, clinical trial results, and regulatory documents specific to this domain. We'll register it in Unity catalog. Then we'll build a knowledge assistant and refine it with additional context and feedback. At that point, this agent can be used by clinical scientists to ask questions about clinical trial protocols.
0:36 Architecture: Agent Bricks Knowledge Assistant
This agent can be used as or as part of multi- aent system. Speaking of data, let's first check out the data sets we have today. I'll be working with three publicly available data sets. Let's first go to Unity catalog, my catalog schema. And the first data source we'll look at is a table containing metadata about clinical trials. Besides general metadata about um about clinical trial, it contains a link to the document and a clinical trial protocol in form of a text column.
1:06 Data Foundation: Protocols & Regulations
Some trials in this data set are still ongoing while the rest are completed. For the completed trials, we have a PDF documents containing the results. We store these documents in a Unity catalog volume. One thing to mention here is that not all PDF documents have the same structure, which can be very challenging if you want to process this data programmatically. In a few minutes, we'll see how Asian bricks can help us.
Here we also have a set of regulatory documents that will use to see if the protocol uh is designed according to the regulatory requirements or if any amendments are needed. You can open documents directly in Unity catalog without the need of downloading it, checking it and potentially uploading back. Let's jump and build our knowledge assistant. For that we go to agents and select our knowledge assistant. You need to provide a name description and then we'll select the sources that we just discussed.
2:26 Setup: Building the Knowledge Assistant
Unity catalog volume. I go to my um catalog schema documents add. You need to specify a name and description of this content. We'll do the same for uh regulatory documents that we have. And now I would like to add a actual clinical trial protocols. As you remember a clinical trial protocol is stored as a column in a metadata table. To be able to add it as a source for our knowledge assistant, we need to create a vector search index on top of it.
I created one in advance. If you would like to create one on your data, follow the documentation from a datab bricks website which contains step-by-step instructions how you can do it. Now I specify a source, my catalog, schema, my vector search. I do need to provide um columns that contains a URL which is a link column and the columns that contains an actual clinical trial protocol. After that, I can click create and wait until my agent is ready to start asking questions.
Great, my agent is ready. Depends on the size of a data that you selected for your agent, it might take some time to actually process this data and create an agent. Now, let's start asking questions to this agent. We'll start with a generic one and ask you to provide us a list of ongoing clinical trials and a description for each of them. We got our result. One important thing to notice here is that our agent can't really identify which clinical trials are ongoing or completed.
3:52 Live Demo: Querying & Discovery
As you remember, information about if this trial is ongoing or completed is stored in a metadata table which we are not using in this assistant. I would like to help my assistant to understand this information better. For that, I will provide an example to understand what does it mean ongoing clinical trials. In this simplifying example, ongoing clinical trials are those that have a protocol but no corresponding result document.
Of course, it's a simplified example, but let's see if it helps our agent to understand uh what does it mean ongoing clinical trial. We'll ask the same question again. Now asking the same question again, agent bricks instantly identifies all ongoing trials and give us exactly the summaries of what we want. But we don't have to stop on at least trials. So how flexible the knowledge assistant really is? Let's ask it to compare eligibility criteria between trials targeting different uh cancer types.
This question allows to compare multiple trial documents and give an answer. We can ask agent to uncover patterns in adverse events profiles across similar therapies. That allows us to dive deeper to find nonobvious insights in minutes. This question implies deeper analysis of a data system isn't limited to group level analysis either. We can drill into a single document. For instance, ask results for a specific clinical trial.
Great, we got results. You can also see that agent quotes documents where it finds certain information. You can open these documents directly here and get access to them. But let's um combine these results with other clinical trials of the same cancer type or we can combine all sources for more advanced guidance. Like for instance try to understand is there are any deviations for a specific clinical trial protocol and um regulatory documents.
Great. We have some work to do for this specific clinical trial. This kind of crowsource guidelinedriven insights used to require weeks of manual work. Now it happens in minutes thanks to agent bricks. This can be a good foundation for a regulatory agent for instance which can suggest protocol changes for ongoing trials in response to changes in the regulatory documents. Knowledge assistant is a perfect way to wrangle and structure clinical documents.
But what about structured data tables metrics records? That's where agent bricks really shines as part of multi- aent system. You can deploy Genie infrastructure sources, pair it with knowledge assistant for text analysis and add other custom agents to generate code, connect to external platforms and more. And all natively integrated into datab bricks secure, scalable lakehouse platform. In short, agent bricks transform complex data into clear, actionable insights and keep you ahead in any fast-paced domain.
So this is what we've built today. We we got some data related to clinical trials. We registered it in unity catalog. We built a knowledge assistant. Then we provided a feedback to knowledge assistant to better understand our questions and then we tested this uh refined agent to ask specific questions and get answers related to clinical trials. This agent can be also part of bigger AI system. Check it out yourself.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.