Building Enterprise AI: Knowledge Tools on Databricks and RAG

Summary

Unconventional AI built Unintelligence, an enterprise knowledge assistant on the Databricks Data and AI platform that ingests documents from Slack, Google Drive, and other sources, embeds them in a vector database, and answers questions through a unified interface.
MCP endpoints expose the system to multiple access points—Slack, a web UI, and developer IDEs such as Claude and Cursor—so researchers can reach institutional knowledge through whichever tool they already use daily.
The same Databricks infrastructure and RAG architecture powers two additional internal tools: Unqualified, an AI-powered recruiting and candidate screening system, and Unassist, an enterprise task intelligence application.

Building Enterprise AI: Knowledge Tools on Databricks and RAG

Large research teams face an impossible challenge: keeping pace with voluminous knowledge spread across disconnected tools like Slack, Google Drive, and specialized databases. Without unified access to institutional knowledge, teams waste time searching for information and reinvent solutions, creating silos and slowing innovation.
Watch how Unconventional AI built Unintelligence, an enterprise knowledge assistant that ingests documents from multiple sources, embeds them in a vector database, and serves them through a unified interface. Learn how to combine Databricks infrastructure, MCP endpoints, and multi-space architecture to enable teams to access knowledge seamlessly through their preferred tools: Slack, web UI, or IDEs like Claude and Cursor.
🤝

Chapters

FAQs

What is Unintelligence and what problem does it solve?

Unintelligence is an enterprise knowledge assistant built by Unconventional AI that ingests documents from sources like Slack and Google Drive, stores them in a vector database, and answers questions through a unified interface. It was created to help staff—particularly non-technical employees—keep pace with the high volume of research papers and discussions shared internally without having to read everything in real time.

How does Unintelligence integrate with tools like Slack and developer IDEs?

Unintelligence exposes MCP endpoints that allow it to be called from multiple interfaces, including Slack, a web UI, and developer IDEs such as Claude and Cursor. This video explains that supporting multiple access points was a key design goal so each team member can interact with institutional knowledge through whichever tool they already use in their daily workflow.

What Databricks infrastructure powers the Unintelligence system?

Unconventional AI uses the Databricks Data and AI platform as the foundation for Unintelligence, leveraging Databricks tooling for data ingestion, vector storage, and serving. This video describes a multi-space architecture that enables teams to manage access control across different knowledge domains within the same platform.

What other AI tools did Unconventional AI build alongside Unintelligence?

In addition to Unintelligence, Unconventional AI built Unqualified, an AI-powered tool for recruiting and candidate screening, and Unassist, an enterprise task intelligence system. This video demonstrates how the same Databricks infrastructure and architectural patterns used for Unintelligence were reused to build these additional internal applications.

Full transcript

[00:08] Hello Bricksters. I'm excited to be here. Nice to meet you all. I'm Ali Asvahanin, uh, CFO of Unconventional AI. If you don't have your headphones on, you might want to put them on. Let me Give me a thumbs up if you can hear me. Perfect. Awesome. Thank you so much. All right. I'm here to talk to you guys
[00:25] about, uh, building our internal knowledge tool called Unintelligence. Uh, I'll give you the backstory of how it came about and then go into a little bit of detail about what we built and how we built it. Uh, so I I don't know how much you guys know
[00:40] about Unconventional AI. It's a pretty ambitious company doing a lot of, uh, frontier research trying to reinvent the entire AI stack from chips all the way to algorithms. And I'm one of the only people at the company who doesn't have a PhD. So, when I first joined in January,
[00:57] uh, I was trying to catch up on what the, uh, researchers are working on and how we're going to get there and so on and so forth. And I was reading Slack every day trying to follow the paper trail of research that was being put in there. And soon I realized the volume of research that's being posted inside
[01:12] Slack is almost impossible to catch up with and for me to actually digest and read. So, uh, what I decided to do was write a simple tool that reads the Slack channels, pulls all the files that are being posted, kind of writes a little digest for it, and, uh, summarizes the Slack
[01:29] conversation so I can go in there and read it, uh, on my own time and just, uh, have a high-level understanding of what's going on. I built that. I showed it to one of our engineers, and he was like, "Oh, can you ask it this?" And I asked the question and it gave an answer. He was like, "Oh,
[01:44] this is really cool. Uh, can you actually have it be like more full-fledged and ingest all these channels and, uh, start parsing the papers and put them into a vector database and then, uh, put an MCP for me and, uh, I want to call it from Slack and do this and do that. I'm like, all
[01:59] right, it's easy. I I I I'm I'm the CFO, I'm not a computer engineer. So, I started working on it and uh we are Databricks customer, so I started learning about Databricks and what tools they have available. And basically started going on this
[02:15] journey for the past 4-5 months uh to build this tool in addition to two other tools I built internally. Uh which I'll walk you guys through. So, this first one is called on intelligence. What the app does is uh pretty straightforward. So, it starts ingesting
[02:32] uh documents from multiple different sources, uh Slack files, Jira, Trello, uh Google Drive, or manual uploads. It also uh basically goes in and ingests all the Slack conversations that are happening in these channels to give context to the
[02:47] papers. Then what it does is uh basically puts them into Databricks, uh runs the ingestion pipelines that I've custom built for this. So, it parses and extracts each of the papers, creates markdowns of the PDFs using Docling and using Databricks
[03:03] infrastructure with the GPUs that we have available. It chunks them and then embeds them into a vector database. And then uh on top of that, I'm running uh an app UI interface. Uh there's a Slack bot that has access to these databases and the models, and then
[03:20] there's MCP endpoints that automatically get provisioned that people can call from inside their preferred AI tool, whether it's uh Claude or uh Codex or Cursor or whatever it is. Uh I'm actually doing all of this on Databricks. So, the app server runs on
[03:35] Databricks. I'm using authentication from Databricks. Uh I'm using their Unity Catalog, using all their model endpoints. So, when I call GPT or Opus, it's all uh from their tools. Uh and all the ingestion and pipelines and the backend GPUs and stuff, all of it is there.
[03:52] Uh I'll give you the idea of a about the per space infrastructure. So, there's multiple spaces in these apps so you can actually have segregated spaces for engineering documents versus research versus my ops kind of stuff that has my contracts
[04:09] and NDAs and so on and so forth. And each of them are segregated from each other so they each have a vector database that gets provisioned real time when the user asks for a new space. And it has a hosted MCP inside Databricks which also automatically gets provisioned so the user just copies
[04:24] basically the uh MCP's URL and they can put it in any tool and use it. So, that's just a high-level overview of the tool and how we've built it. Uh I'll give you guys kind of a quick ish demo of what it looks like so you
[04:40] have a better sense of what is the art of the possible with these tools. So, I'm going to escape out of my presentation. Close it down. Okay. So, this is the tool itself. This is our research library right now. So, you're looking at all the papers
[04:56] that have been ingested. There's over 400 papers in here right now. Uh each of these papers when you click on them they have a dedicated page. Uh some of them don't have full text because they're from public links that don't have uh we don't have access to but if you go to these you can see
[05:12] actually the full text that has been downloaded and the markdown that was created. You can see the in the summary you can see the summary of the Slack conversation around it so on and so forth. And then you can actually has a link to open the Slack thread it was taken from. You can download the PDF. If
[05:28] you want to reference this paper in any of your work you can generate a bibliography uh and so on. So, that's the view of the paper library. You can actually upload documents in here directly. You can delete folders. You can add folders. Uh this is a pretty fully functional
[05:44] things on the on on side here. Just to give you a visualization of how all of this all looks internally, so there is this kind of obsidian-like interface in here that actually has the full mapping of all the papers on how they're related to each other and what context they
[05:59] share. You can actually drill down on the different categories and look at what papers we have available. The size of the bubbles is the intensity of the discussion around the paper. So, if I click on one of these, it actually goes and opens it. You can see all the details about the paper and there's multiple ways to access it.
[06:15] So, that's kind of V1 of what I was building around the research library. Uh then I decided, okay, we need this agentic interface so I can talk to it and ask it questions. So, this is a ChatGPT 5.5 endpoints uh inside Databricks
[06:31] that I've written a bunch of custom tools for that are essentially uh uses our libraries to be able to answer questions for me. So, when I switch my space to, for example, Cadence manuals, which is one of the spaces we're using, it adds the Cadence MCAD automatically to this and it references it when I start asking it questions.
[06:48] Uh and uh you can ask it anything pretty much and uh it it'll give you answers. I have different libraries in here regarding our research, Cadence manual, foundry documentation and documentation. These are TSMC manuals, vendor contracts, which are my NDAs and documents and so
[07:05] on and so forth. So, users can essentially go in here and each of these essentially in the library look pretty similar. So, Cadence manuals are here. None of this is private. These are all public. If anyone has used Cadence here, each of these manuals are about 10,000 pages and there's 300 of them. So, for our
[07:21] engineers to be able to read through these is almost impossible. Uh so, what they did was they went in here and they basically requested a space, Cadence manuals, you see that here, so you can request a new space. You give it a name, you give it a description. You pick the folder drive that you want
[07:36] to sync from. You press submit request. Then what happens is the enterprise admin, happens to be me here, gets uh request in here for that space and uh I approve it. Uh and then it starts ingesting. Uh
[07:52] what's pretty cool about the authentication piece is these spaces are not all shared, so you can actually individually specify using Data Bricks' back end if you want it to be a public available to all or you want it to be restricted to members only. So, for example, for my vendor contracts thing that I have, not everyone in the org has
[08:08] access to it. It's a particular space, only these people can call it. So, uh all of that, again, is pretty cool all on uh their uh back end. You can even configure which model endpoints are picked in real time. So, here I can
[08:24] edit what my orchestrator model is. It's GPT-55 right now. Uh for my classification stuff, I'm using 54. For helper, like titles and digests and stuff, I'm using GPT-5 Nano. And for the vector spaces, I'm using this. This is my prompt. I can edit it
[08:39] real time. Uh etc. So, and then it actually has a memory, so it's learning uh in real time from user feedback uh what it's getting. So, if you actually go to all the stuff that it's learned, uh it actually learns from user feedback
[08:54] and it can get better over time. And uh yeah, that's pretty much it. And then I have a real user management system where I can basically provision people to different spaces and so on and so forth. So, that's kind of the overall overview of that here. Then as I said, I can call it inside Slack. So, if I go inside my
[09:10] Slack, I'm able to call on intelligence and ask it a question. I asked it, "What is the latest version of Tempest and Genus docs you have uh you have?" And it basically calls the same uh back end. Uh resolves it using the NCP that's available to the
[09:25] tool in uh on the on the app and it answers back to me based on what's in the library. If I want to continue reading, I can just click this. It'll take me to the same page. It actually has all the details here. So, for longer research research like papers that I was just playing around with,
[09:41] it actually like comes back with really detailed answers with rendered math, and I can actually click on each of these references. It takes me to the paper. It highlights the section that it's referencing. I can open the open the full view here and look at it, etc., etc. So,
[09:58] that's a little bit about this. Then I can do the same thing inside Claude. I basically as have added the MCP here with Cadence stocks into the mix here. So, I asked it the same question, "What version of Tempus is James you have available?" It basically uh used the
[10:14] MCPs to be able to come up with the exact same answer that it gave me inside the interface. So, with this essentially our users can go in here and do their day-to-day tasks much easier using the tools that are available to them and uh are able to
[10:30] access their data real time wherever they're working. So, if they're working inside uh Slack, they can ask questions there. If they're working on the web interface, they can do that. Or if they're in one of their favorite AI tools, Cursor or Codium, they can actually reference all of this real time.
[10:45] Uh we've built a couple of other tools that I won't show here because they have proprietary data in there, but uh there's a uh candidates uh tool called Unqualified, which essentially our recruiting team is great, but they're
[11:00] small. They're and then they did with uh all sorts of uh resumes. I think we had over 2,000 applications in over a month. So, that's impossible to get through them uh in real time. What we're doing is we're pulling all the data out of Greenhouse using Databricks uh and the
[11:17] Greenhouse APIs. We're staging them all in uh Databricks, processing those. Uh then we take each user's uh each candidate's resume, cover letter, we pull out all the links, and then uh enrich the profile, and create a
[11:33] holistic dossier of the candidate called the 360 view. And then what the what the model does is evaluates them against the job description, company description, and then the internal hiring managers
[11:51] kind of requirements that is not public. And it ranks each candidate based on the level of fit on multiple dimensions and makes a recommendation of kind of gives them a grade from F to A+. And what's really cool actually is
[12:08] I told you guys we were ingesting papers here, research papers here. So, there's a bunch of authors on these research papers, and these are papers we care about. Now, the other tool, which is the candidate tool, has applicants. So, it so happens that there's a cross-section of applicants who we have written research papers from who should be very
[12:24] high on our interview list because I already read your paper. I like your paper. I've posted it. Someone else has read it. So, this actually the two of them now kind of cross-link. So, if there is a candidate that we have a research paper from that was discussed in the company,
[12:41] it actually there's a snipe list that surfaces them to the top so they get expedited. I think my computer went off. I don't think you guys need the computer anymore. I'll just shut it down. And the third thing I'm building, which is actually turns out to be a much harder problem
[12:56] than I expected and I think requires a lot of data processing is another tool called Unassist, which is supposed to be my chief of staff, which essentially tracks my tasks between different applications. So, if
[13:12] I'm using Slack to start a conversation with my colleagues and then they follow up on email, they email the vendor, the email sends a the vendor sends a PO, I go into Ramp to do that, then a payment is processed, then there's a DocuSign. There's no way to connect all
[13:28] these tasks together today for someone to be able to follow that this thing that was done today was finished in this other surface uh end days later. So, my latest challenge has been kind of uh processing and normalizing all the data
[13:43] across all these different uh uh applications that we use inside the enterprise and trying to connect them and uh I'm almost there. I would say Bandish can attest to it, but probably about 75 80% there on the data side. What once the data solves side is solved, I think we can actually get it
[14:00] done, but I think I'll just pause there. I think I've spoken a lot. Uh we have a couple of minutes. If anyone wants to shout out a question, I'm more than happy to answer it. I think it's a intimate enough room that I can answer questions, but you guys tell me.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.