Scaling Governed AI Beyond PoCs: AgentBricks and Lakebase in Regulated Environments

Summary

Skills SA, a South Australian vocational training regulator, moved beyond proof-of-concept AI by building a governed multi-agent system using Databricks Lakebase and AgentBricks to automate organizational self-assessment reviews, reducing assessor workload while maintaining full explainability and audit trails.
The architecture stores prompts and scoring rules as governed data in Lakebase rather than hardcoding them in application logic, enabling rapid policy updates without code changes and making the system evolvable as regulatory requirements shift over time.
Three specialized agents handle different compliance tasks in parallel, Lakebase provides sub-millisecond agent state management and audit logging, and the config-as-data approach embodies a core principle for public sector AI: every decision must be explainable, traceable, and accountable.

Scaling Governed AI Beyond PoCs: AgentBricks and Lakebase in Regulated Environments

Moving AI from successful pilots to regulated production systems requires more than just good models. This presentation shares how Skills SA, a South Australian vocational training regulator, built a governed multi-agent system using Databricks Lakebase and AgentBricks to automate organizational self-assessment (OSA) reviews, reducing assessor workload while maintaining transparency and audit trails.
Learn how to design specialized agents for different compliance tasks, store prompts and scoring rules as governed data to enable rapid policy updates without code changes, and embed governance from day one. Discover the Lakebase architecture that provides sub-millisecond performance for agent state management, and how config-as-data approaches make systems evolvable as regulatory requirements change.
🤝

Chapters

FAQs

What challenge did Skills SA face moving AI from proof-of-concept to production?

Skills SA found that isolated AI proofs-of-concept could not address the questions that arise when AI becomes part of real business processes: how to manage workflow state across agent interactions, how multiple agents collaborate, how humans stay involved in decisions, and how to evolve the system as regulatory requirements change without rebuilding from scratch.

What is the config-as-data approach and why did Skills SA adopt it?

Config-as-data means storing prompts, scoring rules, and policy logic as governed records in Lakebase rather than embedding them in application code. When regulatory requirements change, Skills SA's assessors can update the rules in the database without a code deployment, making the system far more evolvable in a regulated environment where policy changes are frequent.

How does Lakebase support the agentic AI system at Skills SA?

Lakebase provides sub-millisecond performance for the agent state management tables that track each assessment's position in the multi-agent workflow. It also stores the audit trail of every agent decision and assessment output, which is essential for a regulated government system where every decision must be traceable and explainable to stakeholders.

What five principles did Skills SA identify for governed production AI?

Based on their experience, Skills SA identified five principles: embed governance from day one rather than adding it later, design for specialization with agents that have clear and bounded responsibilities, build for human oversight at every decision point, store configuration as data to enable policy evolution without code changes, and treat the audit trail as a first-class feature rather than an afterthought.

Full transcript

[00:08] Hello everyone, and thanks for joining us. I hope you're enjoying the summit so far. It's been pretty cool. Hope you got along to Data After Dark last night. Made best of the music. It was great. Had a good time. Um so yeah, my name's Jared. And I'm the Data AI and Integration Lead
[00:23] at Skills SA as part of South Australian Department of State Development. And Skills SA is responsible for overseeing vocational education and training across South Australia, including the regulation and funding of all our training providers.
[00:39] Joining me today is Claudia Krajic, also from Skills SA, who'll be talking us through a demonstration of the solution that we have built. And Danny Wong from Data Bricks, who'll walk us through the architecture and the platform capabilities that underpin it.
[00:55] So we're going to share a real-world public sector use case that moved beyond um AI experimentation into a governed production architecture. We'll show the business problem we were trying to solve, the architecture we built using Lake Basin Agent Bricks, and the live demonstration of the
[01:12] solution, and some of the lessons that we learned along the way. So just quickly, there might be a uh forward-looking statement in this presentation. So let's dive in. So I'd like to start quickly with the principle that has guided our approach
[01:28] at Skills SA in that AI earns trust in government only when every decision is explainable, traceable, and accountable. And in a regulated environment, successful AI use cases aren't just about the model performance. They're not
[01:45] They're about ensuring AI operates within governance frameworks, supports human decision-making, and can be trusted by both the staff that are using it and the public at large. And that's a big part of the story that we're going to be sharing today.
[02:07] So, most organizations have already proven that AI can generate useful outputs through proof-of-concepts. But the real challenge is taking those isolated use cases and embedding them into day-to-day business operations. As soon as AI becomes part of a business process, new questions begin to emerge.
[02:25] How do you manage the workflow state? How do multiple agents collaborate together? And how do humans stay involved in the decision-making process? And also, how do you evolve the system over time without having to completely real built rebuild it from scratch?
[02:42] And that's the gap that we're referring to here. And at Skills SA, we definitely experienced these throughout our journey. So, to make the discussions a bit more concrete, I'd like to use a real public
[02:57] sector example. Organizational self-assessment, or OSA for short, is a compliance process used to assess training providers against a defined set of standards each year. So, we receive thousands of these
[03:13] submissions and there's multiple touch points. And these submissions are quite large, as you'll see. Claudia will show one a bit later. And they contain both structured and unstructured information and large volumes of free text within their responses.
[03:30] Assessors review these submissions and evaluate them against the known standards and determine whether further investigation or follow-up may be required. The process is highly manual and requires significant assessor effort.
[03:45] And more importantly, it relies on people reviewing large volumes of information and applying standards consistently across every submission. As we looked at the process, we saw an opportunity to use AI to support the assessors
[04:01] to help improve consistency and reduce effort while keeping humans fairly in the loop. So, before we dive into the solution, this is what the OSA looked like prior
[04:17] to the solution being implemented. Providers submitted evidence and responses against the predefined set of standards, and assessors reviewed these submissions and evaluated the information provided, identified potential concerns, and
[04:33] determined whether additional clarification or follow-up was required. And the challenge really wasn't in any individual step in the process. It was the cumulative effort involved in reviewing large volumes of structured and unstructured information
[04:49] while trying to apply those standards consistently across every submission by every individual assessor. The assessors spent a significant amount of time locating relevant information about those providers,
[05:04] understanding context, understanding themes across all the um submissions, as well as understanding and comparing the responses against compliance requirements. These were exactly the types of
[05:20] activities where we believe that AI could provide assistance. The question we asked ourselves wasn't whether AI could help, but how we could introduce it into the process while maintaining transparency and human oversight.
[05:36] That question ultimately drove the architecture we're about to walk through. When we decided to design a solution, we quickly realized that it wasn't a single agent problem. We needed an agentic architecture capable of managing workflow state,
[05:52] specialized responsibilities, and govern decision support at scale. And to explain how we approach this challenge, I'll hand over to Danny. Thank you, Jared.
[06:07] Uh good morning, everyone. So, my name is Danny Wong. I'm a lead solution architect from Databricks. So, I will talk a little bit about like some of the design decisions, some of the architecture behind it. So, something behind the scenes. So, uh I guess the first question is
[06:22] like, can I simply just use the existing analytical stack for this agentic use case? Uh of course, probably you know the answer is no. So, um I think your existing analytical stack is perfect for bringing data
[06:38] uh either batch or streaming manner. Uh you can have reporting dashboard on that, build machine learning models on that. That's perfectly fine, and that is fully governed, fully cataloged. This is the bread and butter. This is the foundation. That's super important, so you keep all of it.
[06:54] But for AI agent, it needs something more. So, um say for example, like if you are having complex multi-step execution, you probably want to keep that in um uh systems for like long-term memory.
[07:10] You do not want to add additional latency to that. Like uh you don't want your users to ask a question and wait for like 10 minutes to get an answer. So, uh latency is the key how you can facilitate sub-millisecond transaction reads and writes. And as a public sector agency, uh we
[07:28] would like to have like full auditability as well, like having full audit trail for every single question, answers, decision. That is key. And finally, um we would like to have a determined more determined and state orchestration and
[07:43] at the same time with the right guy in place. So, that's why we have Lake Base and Agent Breaks fill up the gaps. So, this is a high-level view. So, at the bottom you have your governance layer. As I said, that's our bread and butter,
[08:00] super important having you Unity Catalog as your control to control who access what, provide all the lineage, as well as auditability as well. And data lake open source platform, you can basically store the
[08:16] data in your own managed storage environment. You basically do all the heavy lifting, analysis, cleaning data at this layer, which is perfectly fine. And on top of that, we have the Lake Base layer. So, Lake Base is an open source Postgres SQL server. And we are
[08:34] also running our vector database in it as well. We using PG vector because Skill Success AI already got some experience in PG vector, very comfortable with it. So, we picked PG vector on Lake Base for this particular use case. And we are also using Lake
[08:49] Base to store all the agents state, all the long-term memory. So, that when a user come back to a previous conversation, they can continue that conversation, do not have to tell the agent from the start again. And we find also find the branching
[09:05] features in Lake Base super helpful as well, especially in development, in testing. We want to interact with real real data, but at the same time we don't want to mess up the production data. So, what we do is just simply creating a branch on it, do the testing, and then
[09:20] switch it to the production branch. And scale to zero is also super helpful as well because the agent itself is not like super busy 24/7. At idle time, we want to keep that at like minimal keep keep that in minimal cost. So, we can
[09:36] scale down to zero and then skills essay doesn't have to pay it for the compute for like base. And on top we have that orchestration layer, so agent breaks is the key orchestration tools that we use to orchestrate between different sub agents, and
[09:52] you would need like a nice front end as well, so that nice front end we are hosted on data bricks apps. So, um that's basically the architecture. And why do we choose Lakehouse as the agent state layer? So, first of all,
[10:09] Lakehouse itself is built for AI agent. Um the performance is is is is awesome. Like uh the latency itself is talking about sub milliseconds, so we can have point reads and writes uh for agent state, for session memories, for tool results uh pretty quickly. We uh we do
[10:25] not want to add additional latency to uh the entire process, so that's the perfect fit for this use case. And um PostgreSQL also has a pretty rich ecosystem as well, so we are using one of the extension PG vector as the vector
[10:41] database. So, uh the good thing is you are storing the embeddings alongside with your operational state data. So, basically you can perform your similarity search, run your SQL with a single query, so you don't have to manage and maintain a separate uh vector
[10:56] store component. And um the other point is operational agility. So, uh instead of keep changing the code, committing the code, we are taking a more config-driven approach. So, all the prompts, all the
[11:13] scoring rules, all the guardrails are stored in the database itself. So, if we are trying to change a the agent behavior, what we need to do is to just update um a record within the database. That makes it a lot easier from a day-to-day operation standpoint. And as I said,
[11:30] like scale to zero is quite key as well because the agent itself is not uh super busy 24/7 so at idle time we just scale it to zero to help to keep the operational cost low. Finally is the govern and connected layer so it is all still governed by UC.
[11:47] You do not have to build another governance layer. You simply reuse what you have today to to manage both your analytical data and transactional data. And the bi-directional synchronization between lake base and lake house make it
[12:04] easy as well. So I don't have to build separate pipelines to bring data back and forth. Those are managed pipelines already in place that I can use and leverage quite easily.
[12:20] On the orchestration side for this OSA use case is a supervisor agent use case. So we have one supervisor, we have three agents. So the boundary for the agent is quite clear. So we got three different agents all specialized in different tasks and it is all orchestrated by one
[12:38] supervisor. So in terms of testing is quite easy. We just have to test different agents individually and then um do a combined test for the entire systems. And the plan is we are going to expand it to more and more agents. So
[12:54] instead of adding additional supervisor we will keep the same supervisor. We did a little bit of testing so up to 10-15 agents the latency is still quite acceptable. The reason for that is like we do not have to manage additional supervisors or
[13:13] manage separate company agent systems. We can keep it clean and tidy and lean. And to grow it further we have a concept of domain supervisor. So So, of like putting all the sub-agents under the same supervisor, we will break that down
[13:29] into different domains. Each domains will have their own supervisor, and we can have another layer of supervisor, which is the supervisor of supervisor, to manage all these domain supervisors.
[13:46] And it is composable by design, the entire thing. So, as I mentioned, it is easy for us to perform evaluation quality control because they are just like LEGO blocks. So, all these individual modules performing different tasks with the Agent Brick supervisor tie that up together.
[14:02] And the assessors in this particular use case can continue improve the system by providing feedback as well. So, Agent Brick has a feature called agent learning from human feedback. So, we can continuously providing feedback to Agent
[14:17] Brick to make the system better over time. And the platform itself handle all the heavy lifting, so that from QSA standpoint, they can focus on building the domain logic, building the use case instead of handling all these orchestration infrastructure. So,
[14:33] Databricks handle all the heavy lifting like orchestration infrastructure for them. So, in this particular use case, we have three specialized agents. So, there's an operational agent mainly looking to structure operational
[14:49] metadata. So, they are in tabular format, so we are using Genie space or Genie agent for for for this. So, it provide like instant on-demand access to those provider submission status to operational staff. So,
[15:05] um And the other agent that we've got is an analysis agent, so that is a knowledge assistant. So, the purpose for this analysis agent is to extract the themes, extract the insights from unstructured text because those submissions
[15:20] are unstructured in nature. So, analysis agent is trying to extract all the key information from that. And then use vector search to identify the patterns, identify the similarities, so that our assessors can understand all the submission at scale.
[15:37] And finally, we have a decision support agent, that is the scoring agent. So after I get all those information from the operational agent analyze analyze analyze agent, the decision support agent will make the call will make the scoring to see whether
[15:52] the submission is compliant to all those rules, so that we can with this in place, we can help to maintain like a consistent evaluation criteria. Because even if you can ask human to do it, it is quite hard to have like a super consistent evaluation
[16:09] criteria, but with decision support agent, we just baking all those rules, and then it will basically look into all the submissions in a more structured and standardized way.
[16:24] And we are taking this config as data approach, so all those key configurations, say for example, like the agent configuration, agent stays, prompts, they are all storing in leg base. So if we are trying to make changes to the agent behavior or like
[16:39] changing the rules, we just simply update leg base. And this architecture is very scalable as well, because if we want to land new use case, it is more like a cookie cutter template. It is quite easy for us to duplicate or replicate this to
[16:57] new use cases by simply changing the data direct call within leg base. And another thing is is the scoring rules, because the compliance could change the compliance rules can change over time. So if there is a new requirements coming in, instead of like making changes to the code, all we need
[17:14] is to making a record change on leg base, and then the agent will pick that up for new requests. So, there's no pull request required for for for for this particular case. And also standard updates, we are
[17:30] decoupling the policy itself with the implementation. So, basically all the changes are dynamic get from leg base runtime. So, the journey so far, I think we started this journey with skills SA
[17:45] probably around like 18 months ago. So, at that time the best way that we've got is the Mosaic AI agent framework is a code first approach. It's still quite good because if you have complex use case, if you want high customization, you can do whatever you like with the
[18:02] agent framework. And today for this OSA use case, we are using agent breaks to buy the agent. It gives us a lot easier to get started, shorter turnaround time. Instead of like building the agent in days, now we build that in hours. And it
[18:19] is fully declarative. We can easily expand it by adding adding new sub agents to it. And the plan is we will be expanding the agent portfolio. So, I think in skills SA there's a lot of like backlog projects on agents. So, they have a lot
[18:35] in mind, a lot of innovation that they want to implement. So, we will rinse and repeat this architecture for the new rollout. And then the vision is to have the different domain supervisor focus on different business domain areas
[18:51] and we'll have a supervisor of supervisor to look after all this domain supervisor. So, I think the guidance is if you got a new use case, you probably can ask yourself like can knowledge assistant or Genie can solve it. So, if
[19:06] that is that is something that is solvable by knowledge assistant Genie alone, then we can start from there. And if you have a requirement to combining structure and then structure analysis together within agent systems, then you can consider supervisor agent
[19:23] which is our managed supervisor to orchestrate all the agent interactions. If you need full flexibility, highly customizable, then you can go for the custom code the agent framework approach. So the skills essay approach is like they will start
[19:39] low code for whatever use case first. If that is not achievable or they need more flexibility and customization, then they will go down for the code first approach. So with that I will hand over to Claudia to cover the digital show us a demo in
[19:56] action. Okay, thanks Claudia. Um hello, can you hear me okay? Okay, awesome. All right, I'm going to start. Okay, so I would have loved to do a live demo for you, but being in Australia, I wasn't sure about traveling here if I can
[20:11] connect back to our environment and run the demo. So I was just a bit worried. So anyway, I've got a recording and I'll take you through the steps. Um I will pause if I need to add some more information, but um let's dive in.
[20:27] Um so first of all, hopefully this will play soon. Um Click. Okay. Can you move the cursor back to Oh, sorry. Okay. Thank you. Um so first of all, I'm just going to introduce you to
[20:42] the assessment. So this is the OSA assessment that we talked about in the session. Um it is quite lengthy. It has a number of questions. This is what the training organizations have to fill in. It has 40-50 questions. Each question has a
[21:00] guidance which is the criteria that the training organization needs to adhere to and it has the response from the training organization. So, um we we get hundreds and hundreds of these assessments every year and we have
[21:15] assessors reading them and assessing them manually. You can see this can be a very exhaustive process. So, um so we thought with AI we can cut down on that time. Um it just going to go through the
[21:32] assessment. Um so, now I'm going to jump into the demo. Um this is Databricks app. Um it's built um just using Databricks basically the apps in Databricks and I'm
[21:47] just going to run a simple question, how can you help me? So, um it will come up with three different options. If you cast your mind back to uh the three different agents that we had, we had the assessment agent that will assess each of the um assessment.
[22:04] Uh we have the insights and trends analysis that will extract trends and insights across all assessments and we have the risk and performance analysis as well and data reporting which is Genie sitting over structured data which will answer things like uh how many
[22:21] submissions was the status of of those submissions. Now, let's go into our next question, can you assess assessment two? So, what I showed you before that was assessment two. I'm keeping it to a range of question one to 214 just because it's quite long. We can see here
[22:38] that it is using the um the supervisor. It is uh contacting uh the uh assessor agent. Um so, this is the agent that is meant to analyze those answers against the criteria and then
[22:53] give us a rating. So, you can see it will it will go through each question um and give us a rating of met, not met, partially met based on the criteria that we have put in in in the system prompt of that worker agent.
[23:08] Uh it also has some other information and this can all be customized through the system prompt. Things like uh evidence from the RTO as well as um comments and and reasoning and why it has given that rating so that the assessors can go back and check and
[23:24] validate that uh information. Um just keep in mind that this information is synthetic data um is not real um so it has been done in our in our uh demo environment. Um once the output uh has been received
[23:40] um it's sent to the supervisor which then summarizes everything and presents a bit of an overview to our assessor which is also handy. Now we're going to jump into our next question. Um this is uh this is going to
[23:56] use a different worker agent. So this is going to use we want to know emerging risks across all assessments. So it's going to use our insights or the analysis worker agent which basically looks across all the different uh
[24:11] submissions, assessments and pulls out those emerging risks. Uh you can think that this can be quite a tedious task as well when you have hundreds and hundreds of assessments so it will save time our assessors. The next question is to do with compliance gaps. Again we want our
[24:28] supervisor when we receive that question to direct uh the the question to our insights um worker agent and uh to pull out those compliance gaps that are um that are
[24:43] present in our submissions. And and it does it also gives us um a references to where that information is coming from which again is very good to validate. Um the assessor can go back and validate that information. Uh our next question is uh to do with
[25:00] Genie. This is Genie of a structured uh data. So, this is the operational analysis. So, we just want a simple answer around the status of the submissions. Um and the supervisor, as you can see, sends that data to our third worker agent um
[25:16] and that comes back with basically the submissions um and the status, which is exactly what we asked for. Now, the next step I'm going to switch off to Lake Base. Um so, Lake Base created this Lake Base to
[25:32] basically store all the prompts and the full responses. So, some of these responses uh sorry, some of these prompts that you see here should be familiar. I just random, so it's a what's the status of the submission and the full response is there. This is very handy if you want to
[25:48] do um operational analysis. Um if you like to see the types of questions of prompts your staff is entering uh is all stored within Lake Base. It can be easily accessed and it can be um easily um
[26:03] extracted and analysis performed on it basically. So, we're just going to run through some some of the other prompts I've got there. You can see the responses are there um and that's pretty much it.
[26:19] So, I'm going to switch back to my thing. Yeah. Thank you. Thanks, Coria. Thank you. See if I can get this to work. I think just press next. Yeah.
[26:36] So, yeah. Thank you again. That was great. Um hopefully the demo has made the architecture that we've been talking about feel a little bit more tangible. And what's important is that we didn't build this solution in isolation, but to improve a real operational process.
[26:51] And for our assessors, the goal is to reduce the time spent on routine activities and allow them to focus on the submissions that genuinely required further human judgment, investigation, and or follow-up. For our operational staff, it provides
[27:07] some real-time visibility into workflow statuses and finding bottlenecks and trends across the assessment process. And for Skills to Assess, it helps apply standards more consistently while maintaining complete order trail of how recommendations and decisions were
[27:23] reached. And ultimately, we're not trying to replace the assessors, of course. We're trying to give them better tools to make more informed and consistent decisions across the process. And I can tell you when they were doing this process completely manually, it was
[27:38] incredibly difficult for them to see those types of trends and also the bottlenecks within that process. So, they're hugely grateful for our work. So, while these benefits were valuable for the OSA assessment team, what excited us most was that they weren't
[27:55] unique to this use case. What we found was that the architecture itself creates some operational advantages advantages that can be applied across many regulatory and compliance processes. It allows subject matter experts to focus their attention on the cases that
[28:12] genuinely required their human judgment rather than spending time on a lot of routine review activities. It provides operational teams with that real-time visibility, as I spoke of. And really help them understand those bottlenecks and workload distribution
[28:28] challenges. Because the evaluation criteria and agent behavior are governed centrally, standards can be applied more consistently across those large volumes of submissions. And importantly, every recommendation, decision, and override is traceable,
[28:45] completing completing Sorry, creating an auditable record that supports compliance and accountability. And finally, by treating prompts, scoring rules, and configuration as data, the system can evolve really quickly as policies and standards change
[29:02] without requiring constant redevelopment. So, looking back on our journey, there were a number of lessons that stood out to us. The first was that successful AI initiatives should start with an operational problem at their core.
[29:17] We didn't begin by asking which model we should throw at the problem. We started with a business process that was consuming significant effort and asked how technology could help improve it. The second lesson is that governance can't be bolted on after the fact.
[29:33] In regulated environments, auditability, access controls, explainability, observe and observability need to be designed into the architecture from day one. And the third lesson is that AI systems work best when they're designed to
[29:49] augment people. We found that specialized agents, each with a clearly defined responsibility, were easier to govern, easier to evolve, and far easier for our assessors to trust than a single monolithic agent that was trying to do everything.
[30:08] So, while we've used OSA as the example today, we see it as a reasonable pattern for operational AI in regulated environments. The same characteristics that made OSA a good candidate for AI also exist across many other government enterprise processes.
[30:24] Anywhere you have high volume of information, clearly defined standards, human decision makers, and requirement for governance and auditability, you'll often encounter the same challenges. And what we've shown today is one way of
[30:39] addressing those challenges through a combination of specialized agents, governed operational state, and human oversight. And for us, as Danny alluded, the next step is expanding this pattern into additional domains and workflows. But the broader opportunity is much
[30:55] larger than this single use case, of course. The architecture principles remain the same even as the business problem changes. So, I'd like to finish with five principles that have emerged from our experience building and operating AI
[31:12] solutions in in government. The first is that production AI requires more than just a traditional data platform. Rather than building separate technology stacks, look to extend your existing data platform with operational capabilities that agents need.
[31:33] Keep agents specialized, and wherever possible, treat behavior as data rather than code. We found that storing prompts, configuration, and scoring logic as governed data made the system far easier to evolve and govern over time.
[31:49] Trust isn't a feature, it's actually an architectural outcome. Governance, auditability, observability, and human oversight need to be embedded into the design from the very beginning. And finally, prototype rapidly. Start with the simplest approach that solves the
[32:05] problem and demonstrate value. Low-code capabilities can deliver value quickly, while more advanced patterns can be introduced as the requirements evolve. These principles have shaped our approach to AI at Skills SA, and we
[32:20] believe they're broadly applicable to any organization looking to move from AI experimentation to governed production systems. And so, that brings us to the end of our presentation. We thank you for attending our session, and we hope sharing our
[32:37] journey has provided some practical ideas you can take back to your own organizations. Thanks everyone.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.