Demo: ServiceNow connector from Lakeflow Connect

Summary

• ServiceNow Integration: Lakeflow Connect's ServiceNow connector ingests all ServiceNow data into Databricks with just a few clicks for building high-value use cases.
• RAG Chatbot Demo: Agent Bricks builds a knowledge assistant combining Databricks documentation with ServiceNow Knowledge Base data to provide instant self-service support.
• Hybrid Data Approach: The demo combines vector search for unstructured documentation with structured data queries, handling both document-based and database questions.

Demo: ServiceNow connector from Lakeflow Connect

In this demo, we’ll show you how to build a ServiceNow Knowledge Base Assistant using Lakeflow Connect for data ingestion and Agent Bricks to build the assistant easily. We build an intelligent chatbot to provide instant self-service support on Databricks, powered by our Databricks technical documentation and our ServiceNow Knowledge Base data (kb.databricks.com).
With the new ServiceNow connector, you can easily ingest all of your ServiceNow data into Databricks with a few clicks and build a high-value use case. Lakeflow Connect offers simple and efficient connectors to ingest data from popular applications, databases, cloud storage sources, message buses, and more.

Chapters

• 0:00 Introduction to Lakeflow
• 1:42 Architecture overview
• 2:12 Ingesting ServiceNow data
• 4:48 Data visualization
• 5:44 Creating vector search
• 7:27 Building the AI agent
• 9:45 Conclusion

FAQs

How does Lakeflow Connect simplify ServiceNow data ingestion?

Lakeflow Connect provides a built-in ServiceNow connector that requires just a few clicks to set up. You provide your ServiceNow instance details, and the connector handles all data extraction and incremental updates.

What is shown in this video's demo application?

The demo builds an intelligent chatbot for Databricks support. It combines official Databricks documentation with ServiceNow Knowledge Base data (from kb.databricks.com) to provide instant self-service answers to support questions.

How does the architecture combine multiple data sources for the chatbot?

The demo ingests ServiceNow data using Lakeflow Connect, chunks the data into manageable pieces, creates a vector search index, and uses Agent Bricks to build a question-answering agent over both documentation and knowledge base data.

What other ServiceNow use cases are possible with Lakeflow Connect?

The video mentions other high-value use cases like predicting service incidents before they impact operations and delivering personalized IoT service analytics, all enabled by unified access to ServiceNow data.

Full transcript

0:00 Introduction to Lakeflow

Hi, I'm a solutions architect at datab bricks. We recently launched lake for connect to provide simple built-in data connectors for popular databases, enterprise applications and file sources. In this video, I'll show you how to build a service now knowledgebased assistant using leaf connect for the data ingestion and agent bricks for easily building our assistant. We know that connecting to various data sources can be complex.
So we created Lake for connect to simplify this. With our built-in connectors, data teams can now build efficient incremental pipelines at scale with a very easy setup and maintenance and full integration with the databicks platform, including unified orchestration, observability, and governance. With the new Service Now connector, you can now ingest all of your Service Now data into datab bricks easily in a few clicks and build a high value use cases using your Service Now data.
In this demo, I built an intelligent chatbot to provide instant self-service support on databicks. It is powered by our official database documentation and our service now knowledgebased data that you can also have find on our knowledgebased website kb. databs. com. This is one example of what we can build with service now. But other very high value use cases include predicting service incidents before they impact operations or delivering personalized IoT service analytics.
This is a high-level overview of the architecture used in this demo. First, we'll see how to ingest the service now data using LakeFlow connect. Then, we'll quickly go through a notebook to chunk this data and create a vector search index. Finally, I'll show you how you can set up agent bricks to quickly build a high quality question and answering agent over your knowledge sources. and we will have then the databbook support knowledge assistant ready.

2:12 Ingesting ServiceNow data

The first step to build our assistant is to ingest our service now data. So to do so I can come in here click on add or upload data and what I see here is the list of all the lakeflow connectors available. I can pick the service now connector give a name to my ingestion pipeline. I can choose where I want to store the pipeline events log. So I'll select my catalog and my schema and I can select the connection to the source.
So it has already already been created by my team but I will show you what the connection looks like. What you see here is the information required to connect to my service now instance with the host name and the oath settings so that datab can connect to service now. I can come back here create my pipeline and continue. On this step, we will see a list of all the service now objects that are available for loading and ingesting into data bricks.
So what I can see here are all the service now objects. The ones I'm interested in are the knowledge based data. So I want the feedback table. I also want the use one and the knowledge Then I can choose where I want to store the ingested data in data bricks. So by default it picked the same location as the event log. So I will leave it at as is into my service now schema. This is where I want the data to be. In this last step I can choose the settings for the pipeline.
So I can select a schedule. I can define the frequency at which I want the ingestion to happen for the incremental changes and add notifications to be notified by email whenever there is a failure with the pipeline. And that's it. So on this page, we're now initializing the pipeline. It will react for resources to run it and then set up all my tables and ingest the service now data. So the pipeline is now running. What you can see are the three different tables for the knowledge based data, the feedback, knowledge and used objects.
And you can see the type of table that will be created, the duration of the ingestion and the number of rows that were written. So you can see here that we've upserted 113 rows for the knowledgebased table. I can come into my catalog and quickly show you what the data looks like. So in service now and have a view at the sample data. So what you can see here is the KB knowledge table. You can see the article number in this column and the contents of the article in the text section.

4:48 Data visualization

Now that I have ingested my knowledgebased data, I can also quickly build the dashboard to visualize some of the metrics regarding my knowledge base data. So I can see the number of article by topic, how many are published over time. I can also track the view count and the feedback that were provided by my team. Now that we have ingested our knowledgebased data, we need to clean it and prepare it for our assistant. So what I did with this notebook is create a vector search index that I can provide to the assistant.

5:44 Creating vector search

A vector search allows us to find semantically similar content rather than just exact keyword matches. And this is particularly useful for a chatbot assistant because users will not always frame the question exactly the same way. and the article is written and a semantic search will be much more efficient. So we have three main steps for this process. The first one is to split the knowledgebased articles into smaller chunks of text.
Then we can compute the embeddings which are a numerical representation of the of the article content and finally we can create the vector search index. What I will do rather than go through the code is show you the different steps of the process in my Unity catalog. So I can show you the chunk value table in some simple data. I have the article number. When the article is long, it's split into multiple chunks so that then the agent can retrieve efficiently which trunk has the right information for my user's question.
And then I generated the vector search index via codes. But I can also create it via the UI and select which is the primary key to use the columns to sync the embedding model that I want to use for embedding my text into a numerical value and then how I want to sync this vector search endpoint. So now that my vector search index is ready, I can go and proceed to build my agent. To do so, I will use agent bricks which is our new offering to automatically build and optimize an agent.

7:27 Building the AI agent

So I'll pick the right brick that I want. So knowledge assistance in my case. So to gain some time, I've prepared the config. What I do is just give a description of what I want the agents to be able to do and configure the different knowledge sources to use. So I have first the service now knowledge base data provided in the vector search index. I just said where is the column containing the article reference and the content describing the content quickly and I did exactly the same for the data books documentation that I extracted and chunked and embedded into a vector search index.
What you can do as well if you have other knowledge sources is that you can provide directly PDFs files in a UC volume. um that is also something possible and then I agent bricks will automatically create an evaluation benchmark optimize it and find the right balance between cost and quality so that I don't have to spend too much time developing my own agent with added complexity can take some time to deploy the agent and optimize it so what I've done is deploy it already so that I can show you directly how we can use it Now, so I can ask it a question.
For example, what is Unity catalog? You can see the response with footnotes with links directly to the documentation that was used to provide the answer. But what if I ask something a bit more specific? For example, how can I list all the available tables and their source formats in Unity Catalog? With this one, what we can see is that it provided a query, but also in the footnotes, it linked this time a knowledgebased article rather than the websites with the official documentation.
And if I want to go check this article, uh I can improve my agent by providing a link. But I actually pulled it out here. And you can see this is from the knowledgebased data that we had made available. So we have seen how we can very easily create a knowledge assistance on our enterprise data and you have also seen how you can ingest service now data using our new connector to bring your knowledgebased articles. You can now ingest very easily your service now data but also much more using lera connect and build knowledge assistance chatbots or other high value use cases.

9:45 Conclusion

So get started today with Connect and go to datarig. com/data-ingestion.

Learn more about the Databricks Data and AI platform.