Sponsored by: Quest Software | From Logical Model to Databricks: Automating Data Product Publishing

Summary

CPF Board of Singapore built AS Portal, an agentic AI solution powered by Genie and a large language model, that allows users to query CPF member statistics through natural language without requiring technical expertise.
A routing layer directs queries to either a data agent (Genie) for structured statistics or a document agent (LLM) for document-based questions, with a privilege-scoped token exchange mechanism preventing privilege escalation across multi-user deployments.
A decade-long data transformation journey — from mainframe systems in 1963 to a modern Unified Data Platform on Azure Databricks with bronze, silver, and gold layers — provides the clean, governed data foundation that makes AS Portal's AI accuracy possible.

Chapters

FAQs

What is AS Portal and what does it do for CPF Board?

AS Portal is CPF Board's agentic AI solution that allows users to query CPF member statistics and citizen service information through natural language, with no technical expertise required. It is powered by Genie for structured data queries and an LLM for document-based questions, with a routing layer that directs each query to the appropriate agent.

How does AS Portal route queries between the data agent and the document agent?

AS Portal uses intelligent routing logic to classify incoming questions: data queries such as requests for member statistics are routed to the Genie data agent, while document-based questions are routed to the LLM. This dual-agent architecture ensures that each query type is handled by the component best suited to answer it accurately.

How does CPF Board prevent privilege escalation in multi-user Genie deployments?

CPF Board implemented a privilege-scoped token exchange mechanism so that each user's Genie session operates with only the data access privileges that user is entitled to, rather than a shared elevated credential. This prevents one user from inadvertently or maliciously accessing data they are not authorized to see through the Genie interface.

What data infrastructure did CPF Board build over the past decade to enable AS Portal?

CPF Board evolved from a mainframe-based statistical database through a data warehouse to a modern Unified Data Platform on Azure Databricks with bronze, silver, and gold data layers, encryption, tokenization, and automated ingestion from key business systems. This decade-long foundation — now serving 35-plus departments — is what makes the AI accuracy of AS Portal possible, as it provides the clean, governed data the agents query.

Full transcript

[00:09] Hi, good afternoon. I'm Benedict. Together with me is Alex and Jenning. We're from the Central Province Fund of Singapore. Right here we're here today to share with you about our project as portal. Right. In this presentation,
[00:22] there are some forwardlooking statements. Um remember to take your do the survey at the end of it. Right. Ask portal is our agentic AI solution
[00:35] powered by Genie and NLLLM. At its core, as portal as something transformative. It puts the power of data into the hands of every user. No technical expertise is
[00:47] required. We did not arrive here by accident. It is a result of intentional forwardthinking investment in our data capabilities. Step by step, we moved
[00:59] from silo reporting to integrated analytics and now we are pushing the realm of predictive AIdriven insights. In other words, we've gone through from looking at the rear
[01:12] view mirror into charting the road ahead. We'll start with a video of our journey over the last decade or so.
[01:28] Data has been a key part of our work at CPF board. As we launched the unified data platform or UDP phase 2, let's travel back in time to see how far we have come. The first mainframe computer was installed in CPF board in 1963 to
[01:43] automate the manual ledger accounting system. We used punch cards to update the mainframe computer and keep track of CPF members latest contributions.
[01:56] We introduced the statistical information database or SIB in 1995 to help us generate important statistics. SIB helped us standardize statistical definitions and
[02:08] consolidated view of data in nine subject areas and access to some key business data. SIB was a foundational building block of our data journey. But to support more complex statistical
[02:20] analysis for policy reviews and operations, we launched the data
[02:32] most of our key business data and allowed us
[02:45] has been used by 35 departments and had 10 times more users than SIB. With advancements in cloud computing, big data processing and data science, we embarked on the unified data platform to
[02:58] grow CPF boards capabilities in extracting insights from data. UDP seeks to empower CPF board users by democratizing data and tools. Users can search for data easily, exploit data to
[03:12] generate insights and deploy machine learning models for operations. >> Right? So over a decade ago, we made a big bold step. We were when we were talking about
[03:26] digital transformation, we placed data exploration at the heart of our transformation roadmap. It wasn't a site initiative. It was a strategic commitment from the foundation. We set
[03:39] ourself an ambitious goal to become a truly data smart organization. It was about collecting more data, wielding its purpose and making data
[03:51] accessible yet secure, building comprehensive and highquality data sets, developing worldclass data scientist capability. Data is now woven into every decision
[04:04] that we make at the organization level. To be honest, there were a lot of roadblocks when we started this journey. Our data lived in
[04:18] silos scattered around multiple systems. Pulling together a single coherent view was painful and slow. On top of that, we had no single unified platform.
[04:32] Everyone was working on different tools, duplicating efforts, and reinventing the wheels. And when we wanted to push into advanced territory, machine learning,
[04:44] predictive modeling, we hit a wall. The com the individual laptops just couldn't handle it. It keep crashing. We wanted our people to use data, but
[04:56] the infrastructure was standing in their way. Something had to change. And Alex will cover what we did and how we did it.
[05:25] Thank you, Ben. Yeah, if you ever felt the pain of fragmented data coupled with silo systems, you know exactly what we were actually facing with. We were actually not looking for a quick patch. We wanted a paradigm shift. Moving away
[05:37] from localized restricted hardware was the only way to overcome our historical challenges. And that's why we choose to build on the cloud. First, we wanted a one-stop platform. We
[05:51] created a central space where users can easily assess, visualize, and understand our core business data without jumping through tools and data sources.
[06:03] Second, the ability to scale. Our platform adopts adapts to diverse requirements scaling both vertically as well as horizontally to cater for
[06:15] everything as simple as just reading a simple data to generative AI workloads. Thirdly, a robust ML framework. We enforce proper governance across an
[06:27] automated machine learning life cycle completely completely eliminating the nightmare of bespoke model hosting as well as environment setup. Finally, in
[06:39] terms of our platform road map, future proving is crucial. We do not want to be locked into a rigid specifications, versions or standards. Instead, we wanted our platform to ma
[06:53] remain at the forefront of technology. The product team focuses on the technology and we focus on solving our internal use cases problems.
[07:08] Next, I'll be sharing on the system architecture on how we actually set it up. Our UDP platform unified data platform is built entirely on the Microsoft Azure tech stack align closely with our government road maps. It
[07:22] handles incredibly diverse data whether it's actually legacy mainframe files, flat files or modern JSON files via SFTP as well as API
[07:36] all secured with both encryption and decryption. Looking at our key components, Azure data factory serves as our core orchestration engine triggering data bricks notebook using our own
[07:49] in-houseuilt injection framework. We couple logic apps along with Azure Sentinel to run continuous monitoring and alert notification while pushing our
[08:06] core analytics to PowerBI for dashboarding as well as Neo4j for graph relational use cases. To keep the to keep this massive operation safe, we integrate a custom tokenization
[08:19] service right into the injection pipeline to automatically safeguard sensitive data. And to wrap it all together, we host our machine learning models on the Azure Kubernetative
[08:33] service following the ML ML ops injection standards using Azure API management service to let data bricks talk seamlessly with our on-prem
[08:46] as well as other open systems. Now that you have seen the general landscape of our platform components, let's take a layer deeper, right? To see how the data
[08:58] actually lives, breathes, and transforms across the environment.
[09:10] Okay, this is going to be quite heavy. Let me walk you through how the data actually moves. Most of it originates from our non our on-rem systems. The moment actually it arrives, we handle any necessary decryption and
[09:22] tokenization automatically. The data then flows through our material architecture landing first before passing through the bronze, silver and gold layers. Because security and
[09:34] authorization are our top priorities, sensitive data is enc reenrypted before even it hits the bronze layer.
[09:48] When authorized users need to explore this data in datab bricks, we provide secure views that dynamic dynamically decrypt on the fly when they query for it. To help our users navigate the
[10:01] environment, all our data dictionaries reside in Azure perview for easy exploration. Meanwhile, the actual assess is strictly managed through the Azure entra ID groups which
[10:14] synced directly into data bricks. I know this last three slides were quite technical but the but the architecture is the essential foundation. Ben will be sharing on the tangible real life
[10:27] problems that our platform actually managed to solve. Over to you Ben. >> Thank you Alex. So what happened when we when we had Azure
[10:39] data bricks in place the impact was immediate and it was real right I'm going to talk you through three examples of what did it mean to us creating tangible value for our citizens
[10:51] empowering our staff capability that they never had before first our contact centers I think everyone's very familiar with contact centers right we receive more than half a million calls a year in
[11:03] multiple languages languages serving an incredible diverse citizen base and multiple schemes. The scale alone is staggering to us. The question was how do we put the right people in the right
[11:15] seats at the right time when demand swings wildly and unpredictably. So we built a forecasting model. It does not just predict call volume. It
[11:27] simulates questions across the different scenarios. Think of it as a crystal ball for our predict or for our operations. Instead of reacting to surges when it happens,
[11:39] we now anticipate and deploy resources before the phone starts ringing. As a result of this optimized staff thing, shorter wait times and fundamentally better experience for every member who
[11:53] picks up the phone. The other way that our citizens reach out to us is through our digital platforms. We serve we receive more than million rightins a year through our
[12:07] different channels. It's a massive volume of citizens inquiries that flow in. The challenge was we have a lot of schemes. Members often select the wrong
[12:20] category when writing into us or filling up the forms. What this means to us is that a lot of the inquiries get routed wrongly and our staff spend a lot of time
[12:33] manually redirecting them and the questions take a longer time to be responded to. As a result of this, we built a machine learning model with MLOps, train it on the data bricks,
[12:47] deployed on AKS and now the moment an inquiry comes in, a real time API calls and routes it to the correct department and this has cut wrongly routed cases by
[13:00] a third, meaning that responses and smoother experience for the people we serve. The third example is closer to me because we built a recommenders for our
[13:15] staff. We have a wealth of learning resources that our staff have gone through based on the different criteria like your different um job scope, your seniority
[13:28] and your progression pathway. Previously staff have to go into this wealth and search manually. What we did was we built a personalized
[13:40] learning solution combining UDP with Neo4j knowledge graph. We pulled data from multiple sources like SAP and the offline training programs that we run.
[13:52] Apply machine learning to the map to map the relationships between the skills roles and causes. We then layer an LM powered chatbot on top of that.
[14:04] So now staff would just have to type in what's next for me. They get a tailored recommendation based on their role, skills and growth trajectory.
[14:16] It bec it is a personalized learning at scale. Now we make sure that every person in our organization stays sharp and future ready.
[14:28] Right. So what we have done today is this first uh we have moved from a fragmented siloed data to a unified data platform. It has answered the questions
[14:40] that we raised earlier. Now what this is a transformation we are proud of. We are not done yet. We are just getting started because of how
[14:53] technology has evolved and what is available to us. Jenny will break us through what's our next step.
[15:10] >> Yeah. So, we have come a long way, but we realize a simple truth. Giving people access to data doesn't mean they can use it. We found two barriers. First is data context. We can give someone access to a
[15:22] table, but strange columns names means nothing if there is no domain knowledge. Without that knowledge, people spend too much time figing fig figuring out what questions they can ask. The second
[15:34] barrier is coding skills. Even with a great platform, if someone still needs to write complex SQL or Python to get a simple answer, it isn't true self-service. It creates a dependency
[15:47] where business users wait on their more technical colleagues for daily insights, which slows down decision making. Ultimately, our employees had to be either coding expert or schema expert to
[16:00] get a daily answer. To remove this friction, we decided to explore agentic AI. This drove us to build portal. Our goal is to give our users insight on demand,
[16:14] not just raw tables. To bridge the data context gap, we use rack to retrieve the relevant business terms and domain knowledge behind the scenes. The tool interprets the users intent and fetches
[16:28] the required domain knowledge. So anyone can start asking questions without feeling lost. And to bridge the coding skills gap, users simply ask questions in natural language and ask portal
[16:40] translate them into data queries. Users do not need coding language for daily discovery. Our ultimate vision is to allow users to get insights across
[16:52] different business areas using just natural language to show you how this looks. Here is a quick video demo.
[17:04] US portal is a unified agentic AI solution that lets our users converse with policies, discover data and generate stats all from a single checkbox without coding. First, let's
[17:16] see how someone can ask about our UDP policies and pro processes. For example, if a user doesn't have access to UDP, we can just type how do I onboard to UDP?
[17:30] While the agent is thinking what is actually doing behind the scenes is searching through our process documents. It finds the right document and section of the document extracts that information and outlines the exact steps
[17:44] clearly for the user. Next, what if a user already has assess and wants basic steps? We can use the same check box. In this second example, the question is how
[17:56] many CPF members are there as at December 2025. Ask portal now acts like a data assistant. It reasons about the query, looks into our data catalog to fetch the
[18:08] relevant tables and columns and generates the statistics. Finally, we can easily follow up with a conversational question like what is their total net balance? The agent
[18:21] remembers the context of our conversation without retyping all the filters just now. It realizes that the refers back to the exact same group of members that we found out just now. So
[18:34] it simply looks for the net balance data column and generates the final number for our users. This means they can just check naturally to slice and dice the data. As we can see from the video, all
[18:47] three of these different queries from answering process questions to executing data calculations are handled seamlessly inside this same chat.
[19:00] While the user experience is a simple conversation, here is how the workflow behaves behind the scenes. When a user enters a query, ask portal interprets and routes it to one of the
[19:12] two specialized agents for UDP policy or process questions. The query goes to the wiki agent. This agent is connected to our vector store that contains our wiki documents on policies and processes.
[19:27] And if uh we are quering on numbers and statistics, the query will go to the data agent. This agent is connected to our data catalog vector store and it also uses databicks genie tool to
[19:40] execute the data query. Once the chosen agent retrieves the information, the system generates a clear response in plain English back to the user. To handle database queries smoothly, our
[19:54] data agent uses datab bricks genie spaces. It gave us two major benefits out of the box. The the first benefit is AIBI intelligence. Genie comes with a
[20:06] built-in knowledge store where we can store our synonyms, column descriptions, and example SQL queries. It uses this documentation to translate plain English into accurate data answers, saving us
[20:20] from having to build this translation layer ourselves. The second benefit is governance. Genie integrates natively with Unity catalog UC which means all our existing data
[20:32] access rights and security rules are automatically inherited and respected by the AI agent. This is important for us because we handle data with varying
[20:44] levels of sensitivity and ensuring that restricted information stays secure is a core requirement.
[20:56] To see this in action, let's use this example query. What was the total number of active employment CPF members as December 2025? First, the main agent reads the question and realizes the user wants numbers and
[21:09] statistics. So, it routes the request to the data agent. Next, the data agent maps the query to targeted genie spaces built for specific topics, in this case,
[21:21] member demographics. By routing the request this way, the system doesn't try to query our entire database at once. It finds the right genie space and focuses its execution within those specific
[21:35] tables. This makes the final answer more accurate for our users. Next, Alex will bring us through the technical aspects.
[21:48] >> Thank you, Jaling. Okay, so when we decided to integrate Agentic AI into our unified data platform, we knew we couldn't just slap a large language model into our existing database or platform. In a strict
[22:01] government environment, security and governance are non-negotiable. To empower our platform with AI safely, we built our solution as portto as shared earlier upon these three
[22:14] foundational pillars interface, orchestration and compliance. We designed the three entirely isolated independently managed layers.
[22:26] engine, and the data context are separate are completely separated. This isn't just a performance. is a deliberate architecture choice to ensure strict compliance where every prom and data
[22:40] retrieval action is governed by absolute access control. Okay, I'll be covering on the dry stuff. Just bear with me for a while. Let's look at how this actually maps out to our existing Azure infrastructure. We
[22:54] have evolved our existing data bricks foundation into a highly secure multi-vnet environment. VNET stands for virtual environment. In the top left hand corner, VNET A we have our front
[23:08] end subnet hosted hosting the R UDP web as UDP web front end within a secure app service environment asse. This is the
[23:21] isolated entry point for all our users in VNETB our back and subnet. The heavy lifting happens here. We use Azure Kubernetes service to host our ask
[23:33] portal port containers and Azure keyboard for securing secrets. In VNET C, the request crosses into the datab bricks host and container subnets. This
[23:45] is where the datab bricks genie workspace leaves interfacing natively with our unity catalog UC and SQL warehouses to safely query the data and
[23:57] metadata. Finally, we maintain a sep separate V-Net D specifically for Postgress SQL for state management as well as contextual database management. By
[24:10] compartmentalizing these tiers, the application tier from the database and Gen tier, we ensure that a compromise in any of these layers do not bridge into our core system data assets.
[24:26] Now isolating network is great but there's not a lot of people who's looking it but how do we actually manage our call or how do we manage the user identity when our agent is making the
[24:39] query on the user's behalf how do we prevent privilege escalation priv spill over and the answer for it is actually using the on behalf of or OBO which
[24:52] involves token swap mechanism Here is the flow for phase one and phase two. The user logs in via the single sign on page.
[25:04] Microsoft entra ID authenticates them and issues token A. This token is strictly scoped only for the ask UDP front end application. When phase three, when the user asks the
[25:18] AI for a question that requires that request moves into our internal application
[25:33] context token swap to get token B, the token specifically scope for data bricks. Phase four, which is a final phase. The ask portal process takes this data
[25:46] bricks scope token B and forwards it as a bearer token to the datab bricks genie API. Because datab bricks validates this specific token B natively against the Unity catalog, the AI is constrained
[26:00] entirely by the user's actual true identity and permissions. The AI can only see and query what the specific user is authorized to see. As such, we
[26:12] achieved natural language querying without compromising a single aspect of our existing role-based assess control design. Next, Jenny will be sharing on the lessons and the learning experiences
[26:24] that we have gone along the way. >> Thank you, Alex. Next, we want to share a few lessons we learned while building portal with data bricks genie spaces.
[26:38] Our first takeaway is that meta data is everything. While we already had a data like catalog in place, we found many of our existing definitions too vague for
[26:50] an AI. Human analysts have an advantage. They use their intuition and prior business knowledge to fill in the blanks. and AI doesn't have that implicit context. When we spin up a
[27:03] genie space, we are essentially onboarding a new automated team member that needs everything spelled out clearly. So, we went back to refine our definitions inside UC. First, we focused
[27:16] on writing descriptive clear commands for tables and columns. Second, we translated cryptic database abbrevations into plain business language. This reduces hallucinations and ensures Genie
[27:29] pulls the relevant data our users are looking for. Our second lesson is to actively teach Genie our business language. Out of the
[27:41] box, LLM doesn't know our internal acronyms, unique rules, or how our tables link together. To bridge this, we explicitly map our tables relationships and join paths ahead of time. This lets
[27:55] Genie safely navigate across tables without having to guess. We also predefine our filters and calculations as reusable SQL expressions. My by
[28:07] mapping everyday business terms like active members directly to these code snippets. We created a reliable bridge.
[28:20] We also provided a library of example SQL queries for common questions. While the expressions we just looked at at the previous slide, they handle individual filters and calculations. These example queries show Genie the entire end to end
[28:33] query structure. They show Genie the exact logics, join patterns, and perimeter structures to replicate. When we first started, we wrote long paragraphs of text instructions to
[28:46] explain how our schemas work. But we found that extract text prompts they easily misinterpreted. We now keep text instructions light reserving them for general behavioral
[28:58] rules. For data logic, giving Genie concrete SQL examples to follow is more reliable. Our final lesson is about development strategy. Build, measure, optimize, and
[29:13] repeat. When we first saw Genie in action, we were tempted to connect all other tables in our data warehouse at once. We learned that handling everything at once makes debugging overwhelming. We adopted a step
[29:27] approach. We started small by focusing only on our most requested matrix from single tables. This gave us a stable, reliable foundation. Only after we were
[29:39] confident in single table accuracy did we start introducing multi-table joints. Taking it one step at a time kept our errors manageable and stop our mistakes from compounding.
[29:52] The second part of this loop is continuous benchmarking. We move away from checking a few queries here and there to a more rigorous benchmarking process. We put together a static test
[30:05] suite of golden questions paired with pre-verified SQL answers. When a test run drops in accuracy, we isolate and fix one variable at a time, either changing a single description or prompt
[30:17] before we retest, ensuring we know exactly what moved the needle. Treating our genie spaces as living continuously reviewed environments took some patience, but it's what helped us
[30:30] transform our genie spaces into a dependent tool for our users. This brings us to the end of our presentation and to the core mission behind us portal. Our goal wasn't just
[30:43] to deploy a new AI tool. It was to fundamentally change how we work. transforming data from a technical asset into an everyday capability for everyone. Thank you for your time and we
[30:55] are happy to take any questions at the site.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.