Databricks Agents in Production: Caltrans Traffic AI at Scale

Summary

Caltrans built a multi-agent AI platform on the Databricks lakehouse to unify fragmented transportation data across California's highway system, reducing manual traffic analysis from 1,000 hours to 2 seconds.
The architecture uses Delta Lake as a governed single source of truth, Lakebase for high-performance operational lookups and spatial queries, and a centralized AI control plane that orchestrates domain agents while enforcing policy guardrails.
The solution has been live in production for eight months, enables incident detection within 20–25 minutes, accelerates new agent deployment from months to days, and embeds governance through Unity Catalog across all agents.

Databricks Agents in Production: Caltrans Traffic AI at Scale

Caltrans manages transportation for 10 terabytes of structured telemetry, unstructured documents, and geospatial data across California's highway system, but fragmented legacy systems prevented unified decision-making. Data that should power proactive safety, congestion management, and incident response was siloed, making it impossible to move from insight to action at statewide scale.
Caltrans built a multi-agent AI platform on the Databricks lakehouse that consolidates all transportation data into a governed Delta Lake foundation, with Lakebase providing high-performance operational lookups and spatial queries for live agent execution. A centralized AI control plane orchestrates domain agents, enforces policy guardrails, and ensures every response is grounded in authoritative data. The platform reduced manual analysis from 1,000 hours to 2 seconds, enables incident detection within 20-25 minutes, and accelerates deployment from months to days while embedding governance through Unity Catalog across all agents.
🤝

Chapters

FAQs

What data challenges did Caltrans face before building the Databricks platform?

Caltrans had critical transportation data — including structured telemetry, unstructured documents, and geospatial data — spread across multiple fragmented legacy systems. This fragmentation made it impossible to move from insight to action at a statewide level, forcing reactive rather than proactive decision-making.

How does Lakebase support real-time operational decisions in the Caltrans platform?

Lakebase provides high-performance operational lookups and spatial queries that agents execute during live decision-making. It handles the low-latency, transactional access patterns needed for real-time agent execution, complementing Delta Lake which serves as the analytical and AI foundation.

What is the AI control plane in Caltrans's multi-agent architecture?

The AI control plane is a centralized orchestration layer that routes tasks to domain-specific agents, enforces policy guardrails, and ensures every agent response is grounded in authoritative data from the governed Delta Lake foundation. It provides a consistent governance layer across all agents deployed in the platform.

How did the Caltrans platform improve incident detection and response times?

By consolidating all transportation data into a unified governed foundation and enabling AI agents to query it in real time, the platform reduced incident detection to within 20–25 minutes. This compares favorably to the prior fragmented approach that could not support timely, data-driven incident response at statewide scale.

Full transcript

[00:07] Good afternoon everybody. Thank you for being with us today. Um my name is Ajali. I am a managing director with Accenture, part of our data and analytics practice. Um I'm joined by Duper. He'll introduce himself very
[00:23] shortly. Um today we will share how one of the largest departments of transportation um is evolving from reactive traffic analytics to an um agent forward mobility intelligence platform. And this solution is built on
[00:39] Databricks Lakehouse and Lake Base. Um the journey started with a very familiar challenge that most um uh people are facing today. This critical transportation data that is spread across multiple different systems. Uh and this data is structured data, it's unstructured data.
[00:56] Um it's geospatial data, which is very significant for transportation. Um making it really difficult for Caltrans, who who we implemented the solution for, uh to move from insight to action at a statewide level. So by creating a
[01:12] unified governed data foundation that we will talk about, um we essentially created a single source of truth for analytics, for AI, as well as for um operational decision-making. So um happy to be here today. And um we will
[01:32] discuss how consolidating onto the Databricks architecture reduced platform complexity. It accelerated access to insights. Duper will cover some of the outcomes from this program. Uh improved query performance, strengthened governance, and created a scalable
[01:47] foundation for uh future AI-driven transportation. So looking forward to sharing what we did on this program. This is solution that has been live in production for about 8 months now. So we'll also share some of the lessons learned as we were on this journey and
[02:04] you know, partnered with Caltrans to shape the future of mobility and transportation operations. So with that I will hand this over to Duber. Thank you. Hey Jelly, I'll thank you Jelly for introduction. First
[02:20] I want to welcome everyone. You make it to the last day of the of the summit, right? You should power yourself, stick around and during the lunch time, thank you for joining us today. So there was a party last night. And the party last night. I I missed that you enjoyed it. So first I want to introduce myself. My name is
[02:37] Duber Tong. I work for Department of Transportation Caltrans. I'm a I'm the division chief for traffic operation. Then you wonder like after this meeting probably you don't remember me, but you will remember we because I'm the AKA your state traffic
[02:54] engineer. When you stuck in the traffic, you will remember me from now on of this this day now. So I'm guy to I'm the guy to blame when you stuck in the traffic basically. So today I'm very lucky have a chance to talk about the general AI application that Caltrans has
[03:10] been just developed not too long ago. We have been in production using now. It's called TMI traffic mobility inside. So So really my job as a traffic engineer for the state is improve the safety. Reduce congestions and respond incident
[03:28] as fast and quick as possible. But as a traffic engineer we love data. Traffic is a lot of data. The volume data, the roadway data tons of data. So I will call ourselves
[03:44] traffic engineer plus engineer. We like collect stuff. I don't know. How many you are engineer, electrical engineer? We never throw away stuff. We always accumulate. So, we just have a lot of things in our place. So, I call our engine is we are rich in
[04:01] data, but poor in insight and analyze. So, how do you solve the problem? We we we are very glad to work with Accenture, the data brick to use the the generative technology to able to bring all the data.
[04:18] They are structured and structured data, old data, new data, live data. So, we can put it in together, make some sense of it. That that will help us to look at any abnormal situation on the highway.
[04:35] To look at how to how to find a way to reduce the congestions. And look at the crash data, how we can improve the safety. And most importantly is safety is the number one priority for our department.
[04:55] And this tool, not just put the data in the platform, it's really help us to make decision faster and better decision that we never had that before. Next slide, please. So, let me go to deep dive on the secret
[05:10] of our data and how complex what's that mean. On your left hand side, the vertical column, you can see all the different type of data. A lot of the acronym, but I just give you really high level. We have the CHP incident data. Incident
[05:26] mean doesn't mean accident. It could be low spill, could be picking up a mattress on the roadside. That could be incident that cause disruption to the traffic. And we have crash data. If a vehicle crash, pedestrian and
[05:42] bicycle. And we have a lot of lane closure data, LCS. One of the acronyms. We close lane a lot, not because we like to, but we have to because we have to do maintain the roadway, doing new construction, and
[05:57] we have permit permitting. People need to improve roadway. We have to let them to access the roadway, then they have to do lane closure to do the work. Weather. Weather is a big part of traffic disruption. Weather days is a very important for us to know. Snowing,
[06:14] raining, high wind. Those are the data we have. So, the left-hand side you can see all different type of data, plus detection. The roadway detector, we have a lot of detector all the state highway to look at
[06:30] how many volume, what kind of vehicle, is it a truck? You have how big the truck you have. Those type of factor for our traffic engineer big decisions. And then bottom of that is engineer. What we like to do?
[06:46] Any Any engineer here? Raise your hand. We like to read what called manual. We like manual. We like guidance. We like operating system, right? So, we do have a lot of the manual. As a state, we have our own state manual. We have a federal
[07:02] manual. We have industry guideline. What do you do? So, in this case, we able to digest up to 1,200 manual and guidance. So, the manual we talk about is not like
[07:17] 10 pages. So, the manual is like this gentleman the lunch box, that big. Each manual could be 1,500 to 2,000 pages. So, this is how much information we need to digest.
[07:33] And then we able to use this uniform platform with open AI and actual technology to put all those data platform together. With With this data we up to like 10 terabyte of data and up to
[07:49] 66 billion records. So, I did a quick check for 10 terabyte of data if I convert it to pages it's about 1 billion pages.
[08:06] So, give you give a guess, ask you guys. If I stack up 1 billion pages, how tall that you think? Any guess?
[08:22] Equivalent to 20 Eiffel Towers. So, basically we ask our engineer with 20 Eiffel Tower information, make some sense of it. Good luck. So, this tools is really help us put it together and look at thing. A few example on the the right hand side to
[08:38] slide to show like one of the report we generate called congestion report. This report we used to take close to 1,000 staff hours to prepare the report.
[08:53] Now the report how What do you think how long it take for me to do this report? Used to be take 1,000 hour with the AI. Anyway guess? 10 minutes?
[09:10] By the time I say how long? 2 seconds. 2 seconds. That's all it take to generate that congestion report used to take 1,000 staff hours to do it. That's how fast and how good that is. And of of course, back to the incident. Incident is one of key thing we
[09:26] we rely on the CHP data, but it's delay. It's hard to see all the incident. With these tools I can see all the incident happen any corner in state I state of California state highway system, when they happen within 20 minutes 25
[09:43] minutes, sorry, I can see that in the tools. I can manage it. I can dispatch it. I can do something about it. And lastly, as I mentioned the 1,200 pages document that we have, it would help us
[09:59] create the knowledge bank. The knowledge bank is help us to get insight faster. Just give it one example. I use that tools. Uh I have one simple engineering question I need to answer. Used to it would take me 2 hours to look at engineering document,
[10:15] manual, guidance to respond the email. Now, I just count yesterday, it take me less than 10 minutes to get the insight and write the email, get back to to the people. So, this is the power of the GenAI. I want to
[10:31] showcase that to you today. Now, I want to pass the time at a jelly to essential to get to deep dive the technology. Again, I'm not I'm not a technology guy. I'm just engineer, job engineer. So, let him talk about that. Thank you. Yeah, and and
[10:47] Duber is our is the business lead for the solution. So, he's been intimately involved with the adoption of the solution. He'll talk about it at the end. So, we'll run through this a little bit quick. Let's just start Let me just start a little bit about just like very quickly
[11:03] the vision of the solution. So, it's not uh super complex here. So, okay. The first pillar we are talking about a unified enterprise data foundation. We talked about it a lot. Lots of data, structured, unstructured, coming in at
[11:18] different cadences. They just needed a single source of truth. That's something we built. Second pillar, scalability. You heard 66 billion records, 8 terabytes, and only growing, right? So, we really needed a solution that could support statewide traffic operations,
[11:33] insights, and analytics. So, definitely scalable. Third, in transportation, the where is absolutely paramount. Like, just just qualitatively understanding incident data without knowing the where is completely pointless. So, the geospatial intelligence that had to be
[11:50] native to our solution, not an afterthought, not a bolt-on. Spatial analysis had was embedded right from the start. Fourth, actionable decision support. There was a lot of unstructured data that was taken in as part of that. So, these are like AASHTO guidelines, state guidelines. Remember, this tool
[12:06] also gives countermeasure recommendations. So, all of that was taken in for an actionable decision support solution. And then just remember, there's lots of ways the user can interact with this. There's like maps, there's dashboards, there's natural language query, but they all query the same govern data foundation
[12:22] giving consistent responses. Fifth, an enterprise solution. It's it's not a point solution. Right now, there are four use cases that are implemented on it, and we are going to incrementally scale this to 17 more use cases. So, this is And it will share that same
[12:39] architecture. And finally, last but not the least, governance. So, uh governance, lineage, security that is built through uh like it was a consideration right from day one uh with centralized controls and policies. Um so, essentially, this govern data
[12:56] foundation serves both the analytical use cases as well as any kind of like um AI use cases. Um let's move through quickly. And we will um Hm. Okay, I'll just cover this, and then we will very briefly also uh play a
[13:13] video of the solution. I can't sit here and do a live demo, but if there's anybody in the audience who's interested, we can certainly take that later. So, one of the architectural decisions that we made fairly early on was separating the user um interaction from the agent execution. So, users
[13:28] interact through natural language, maps, dashboards, reports, but they are not interacting with one particular single agent. They are interacting with what what we are calling a centralized AI control plane, but that's primarily used for doing agent orchestration. Now, the
[13:45] control plane plays a very important role. It is essentially understanding the intent. It's then doing agent routing, it's doing orchestration, it's doing memory management, and then it's also doing tracking. So, if you take a typical question which is on corridor
[14:00] reliability, that might require traffic telemetry data, it might require incident data, it might require geospatial context as well as the transportation guideline documents. So, the control plane will decompose this intent, it routes it to the right domain
[14:17] agent, it lets them collaborate, and then give a response that's grounded in all these data sets and documentation that Duber talked about. Um, the data, and then we'll cut to the chase, right? The execution and data plane on the right, the foundation is on
[14:33] Databricks. The Delta Lake holds the data, Unity Catalog governs it. As of today, the model serving, ML Ops, as well as the MCP solution is currently built on Azure AI Foundry, but we are
[14:49] very much looking in our road map to have that consolidated onto Databricks. So, they come under the same governance. So, if I'll just say two key things to take away, um, every response from any agent is grounded in this governed enterprise
[15:06] data. Uh, and then the second thing is really where the governance lives. We do not implement the governance at the agent level, but we are implementing it through Unity Catalog. So, every agent we have today and every agent we add tomorrow will inherit the
[15:23] same security and lineage. So, this is truly like a a centralized trusted data foundation. Um, I will be very brave and go and show the demo now.
[15:49] Just to give everybody a feel of what it is. Caltrans traffic mobility insights Gen AI platform. We have created an enterprise data and AI platform integrating 10 terabytes of Caltrans roadway data, including traffic, incidents, freight, and transit.
[16:04] Along with insights from nearly 1,500 documents accessible to Caltrans staff, the platform delivers key capabilities across safety analytics, freight management, emergency response, and workforce enablement,
[16:20] transforming how Caltrans manages California's roadways. Our multi-agentic AI platform includes a user-friendly chatbot interface and advanced mobility and safety dashboards, delivering multimodal responses, including interactive maps, charts,
[16:37] tables, summaries, and reports. Alongside real-time traffic data and computer vision capabilities for analyzing CCTV camera feeds. For example, Caltrans aims to eliminate fatalities and serious injuries on
[16:53] roadways. The platform helps identify problem areas and root causes, enabling traffic engineers to analyze incident patterns on roadways and get recommendations to reduce rear-end collisions during peak traffic. California ports are critical for global
[17:10] trade and the state's economy. The platform aids Caltrans in truck permitting and freight routing, enhancing truck safety. For instance, it quickly identifies routes for large freight vehicles across complex road networks, reducing
[17:26] congestion and pollution, while boosting economic productivity and improving safety. The platform identifies optimal evacuation routes during various crises, such as wildfires or earthquakes. For example, in the event of a
[17:42] major earthquake in San Francisco, our tool rapidly generates an emergency evacuation route to the East Bay, accounting for bridge closures. In summary, the platform harnesses cutting-edge GenAI technology to make Caltrans proactive, enabling
[17:59] proactive safety on California's roads and work zones. It streamlines operations, enabling faster onboarding, preserving institutional knowledge, and reducing processes that once took months to just hours, while delivering
[18:15] real impact for California residents, reducing carbon footprint, improving transit reliability, and promoting equitable transportation for all Californians. Okay. So, while my trusted friend here
[18:31] is going to get it back to the presentation. So, um one of the use cases, of course, other than all the congestion planning, etc., is also it's being seriously considered for large event planning. Uh with sometimes potentially to LA 28, as
[18:46] you can see. So, um I will just uh start talking about the Oh, thank you. Uh I'll start talking about the data architecture now. Uh and again, we'll run through this a little bit quickly, but you know, we are available for questions after I I also
[19:01] want to say Rex sitting on top, he's the solution architect for this. So, feel free to um huddle uh back. Okay. So, um the uh Oh, no. Did we Ah, okay. Okay. So, the lakehouse uh is
[19:18] essentially the architectural backbone of our We we call this TMI. Not It's not too much information. It's traffic mobility insight solution. So, it's the architectural backbone of TMI because we need a platform capable of supporting like the analytics as well as the AI use
[19:34] cases. Um now, this is essentially the foundation where all the data sets that Duber was talking about, like the 20 Eiffel Towers, that's that's the worth of data is actually sitting here. And it's data coming in from crash data, incident data, uh geospatial data from LRS as well as
[19:51] transportation knowledge data. So, um and these are all arriving at different um frequencies, as you can see, uh from 25-minute intervals all the way to uh daily uh intervals. And each of these in
[20:07] the before we built our solution had their own separate uh it came with their own like ingestion pipeline, its own storage, and its own governance. So, that's is the first thing that we set out to standardize at the beginning. Uh so, the answer is kind of in the center, the Delta Lake. So, everything lands in
[20:23] the Delta Lake, and it flows through the medallion architecture, bronze, silver, gold, and then Unity Catalog governs all of it. And then lake base is sitting alongside, essentially providing the operational context, which is necessary for the fast operational lookups and
[20:39] spatial workloads, uh depending on what queries the traffic engineer give gives. So, we needed the structured um telemetry data, unstructured documents, geospatial data in the same governed place, and the same tables to serve uh analytics, ML forecasting, geospatial
[20:55] and analytics, as well as live agents. And then we didn't want to copy this data into separate feature stores or analytics smart. So, the lakehouse architecture was very suitable to meet all those needs, and hence we have implemented it and then you can see the rich use cases on the right that
[21:11] Caltrans was able to derive from it whether it's near real-time traffic and incident monitoring and prevention generating the MPR report that's the one Duper spoke about it took him thousand like man-hours earlier but now it's just like way way lesser. So we have some
[21:26] metrics that we measured on those efficiencies then different kinds of speed prediction and congestion forecasting as well as this is an excellent um uh tool to train some of the traffic engineers especially when it comes to like the policies and the
[21:42] um countermeasure recommendations. So Lakehouse it eliminated all our duplicate data movement and it drastically reduced like integration complexity and now we are ready to essentially this platform can scale to statewide.
[21:57] So let's go to the next one. Let's continue with this. So a lesson that we learned very early on is that an agentic architecture magnifies the architectural complexity. So any kind of like unstable pipeline uh any extra like or duplicates of
[22:14] databases every separate governance process it it potentially becomes a liability. So before we went to town adding more agents we took a very deliberate decision to simplify the underlying data architecture and we spent like some some focused time on it.
[22:31] So the architecture became simpler easier to govern and significantly easier to scale through three things that we did. So first we replaced we talked about it all the custom ETLs coming from all those source systems uh we replaced those with native native
[22:46] sync jobs uh from the gold layer into the lake base. So we had fewer moving parts and fewer chances of failure. Um prior to Lakebase operational context and analytics workloads were delivered by two separate architectures, we
[23:03] consolidated those and that essentially helped us reduce a lot of duplication, governance, monitoring, and security overhead that we had. Um and then the second is of course when we consolidated, we consolidated these
[23:20] uh two um analytical and operational architectures into Data Bricks, but we did not lose performance in any way. So, we still had the high performance we needed with like index lookups and spatial queries. Um time is like I think time is of importance here. Like if you
[23:36] think of a traffic engineer, if he has to like make a query, get a response, look at a countermeasure recommendation, time is of essence. So, that's why like we had we just keep repeating that. And the third point is we did it without breaking any compatibility to like the
[23:52] as a system and workflow. So, we we're talking about this in a context where there was an architecture, a less than optimal architecture and we came in and we optimized it on Data Bricks. So, the this third piece now, because Lake Base was able to use uh PostgreSQL
[24:07] extensions, we were able to use uh PostGIS, which is a popular GIS extension to handle the near real-time geospatial calculations. And this was this was huge because again for transportation, geospatial data is of utmost important uh importance. So,
[24:24] that's a little bit about like this particular architecture. And then um it was important enough uh that we wanted a separate discussion just on our geospatial analytics architecture. So, again, transportation is a geospatial
[24:41] problem. Nothing in transportation means anything unless it's attached to a road network. So, whether that's a sensor reading, an incident, or crash. So, we treated geospatial intelligence as a core part of the architecture. So, in this flow, you can see that Databricks
[24:57] performs the large-scale analytical processing. It joins the traffic telemetry data with the Caltrans LRS data, which is their uh linear referencing system. Uh and essentially their road network, and it computes scalable materialized views.
[25:14] Those analytical outputs are then synced with Lake Base, where the post GIS executes the low-latency spatial operations, so that would be segment analysis, ramp and merge diverge detections, or other lane configurations.
[25:30] And on top of that, we have a spatial analytics agent. This dynamically generates uh spatial SQL from a natural language query. A natural language query as simple as show me the merge areas and elevated incident risks on a particular
[25:47] corridor. And ArcGIS portal, server, and JavaScript SDK, they uh render this on an interactive map. So, you can see the various components kind of working together. So, uh two things if if um that that I want to call out. One is uh
[26:05] this takes away um or or rather this brings down the barrier. You no longer need somebody with highly specialized GIS skills to do spatial analysis, and this is very significant. Like, anybody who knows English can through this tool do that assessment. Um and then the
[26:22] second thing is just how the components are working together, the collaboration between Databricks being the intelligence layer and ArcGIS is the visualization layer. Uh and because they are all working off the same governed foundations, the answers never misalign.
[26:40] Okay. Uh let's talk about how we are using Lake Base in our solution. So, this was one of the most interesting design decisions that was um where we introduced Lake Base as the operational layer uh for agent execution. So, um
[26:56] we had analytics requirements, significant analytics requirements, but the the queries to the agents, those were definitely operational workloads. So, what what the agents needed was low latency access to curated trusted
[27:12] information during some of these live interactions with traffic engineers. And we felt that lake base was most suited for that. Um we I I already shared that the gold layer data products within lake house, they are synced into lake base.
[27:27] Uh and almost think of these as like agent ready materialized view. Um so, this really gave us operational performance, but at the same time it maintained the governance consistency. Uh and you know, we heard at the at the keynote yesterday or the day
[27:43] before, one area we are certainly going to you know, look at a deep lane our road map is the lake base transactional application platform and see how that can further improve our lake base solution here. Um very quickly the advantages that we
[27:59] got that it was fast. So, the it was native gold to lake base sync that replaced all the custom ETLs. So, it was quick, it was performant. We retired some of the standalone post SQL while keeping some of the high
[28:15] performance index look up and spatial queries. Um it was native meaning the post GIS libraries ran within lake base. Um of course, it was near real time. I would say it's near real time, but that was more a business decision. That's
[28:30] what Caltrans wanted. So, low late latency reads um really was enabled by the lake base solution. And then of course, it's you know, we can't underscore governance enough. Um lake base is part of data
[28:45] bricks. It lives within the same um unit catalog governance, lineage, and security. And that was very important to Caltrans, right? Like reliability of the solution was very important. So, that's essentially how we are using lake
[29:01] base. And I know I'm running through this a little bit fast. We still have a little bit to cover, and I don't want anyone to miss like the outcomes that Duper is going to share. Um on the how we did the lake base migration, again, like our motivation to move to lake base was twofold. The first
[29:17] was uh really scale, right? Uh we heard 66 billion, and that's only four use cases. So, uh for uh stand-alone PostgreSQL simply could not hold up to that to that volume. Um and
[29:36] um that that ceiling was a real constraint for PostgreSQL uh and what their agents could reliably do in production. Um the second uh reason why we moved was platform consolidation. So, every additional platform that we had that
[29:52] would that caused operational complexity that we did not want. It needed more governance, it needed security, it needed monitoring, patching, staffing. So, that was it it it it was less than uh ideal. So, lake base let us address
[30:08] both of these at once. We moved the operational context into a layer that scales with the lake house. So, it's 66 billion and this architecture can easily scale to much more. Um so, we lifted the performance ceiling and uh ensured that
[30:25] um the uh we lifted the performance ceiling as well as consolidated the platform. So, it was like a double win for us. Um uh just a little bit, we did this over a period of I would say 4 months with a
[30:40] uh qualified like Accenture team. We worked with Caltrans on the testing and, um, you know, we, uh, benchmark first, we built and validated with real, uh, data quality test and then deployed to production. This was deployed to an already live production system. So,
[30:56] there was a lot of care taken in the, in the testing. Um, and then just a little bit on the results. So, we got the results on both fronts, on performance scale and query response. Uh, the times improved by 17% on Lake base native indexing. And on
[31:11] cost and operations, compute dropped about 52% um, and then time to insight improved by about 50%. But, that was primarily because of the native sync between Lakehouse and Lake base. Uh, and then what's not on the slide is, of course, governance got a lot easier and simpler.
[31:27] And we were able to take kind of all this time that we were spending in maintaining multiple platforms and reinvesting back into building like business use cases instead of like the operational overhead of maintaining two platforms. Okay, Lakehouse.
[31:43] I got like 8 minutes to cover. Um, Okay, so, uh, I would say, uh, Lakehouse, this is essentially what's supporting the 66 billion records from 10 sources, uh, and over like 10 terabytes. Um, the size was definitely
[32:00] compelling, but that's not the most, um, uh, interesting part. Um, the architecture was really powerful because it brought together structured telemetry data, geospatial data, and unstructured transportation documents, uh, into one environment. So, traditional
[32:16] architectures might separate this and have like different solutions for a reporting platform, ML platform, document repositories, but we wanted to avoid that from a start to avoid like, uh, you know, multiple, um, copies of data, sync jobs, and potentially, uh,
[32:34] risking drifts in the, in the data. So, we avoided that pattern. So, what this essentially means is forecasting models, dashboards, and retrieval systems, geospatial agents, all other agents they consume from the same governed data set. So, this dramatically
[32:51] reduce data duplication and eliminated any kind of like synchronization challenges. Every workload benefited from the same governance model, the same lineage model model, and the security model. Um yeah, and then I can say that what we
[33:08] built here is truly that one system of record for transportation decision intelligence data because of all of this. And this scales statewide volume uh very efficiently. And we are actually in the process of doing that uh as we
[33:25] speak. Okay, moving on. I'll cover Unity Catalog and how we are using Unity Catalog in our solution. So, um this is essentially the governance engine for all our solution. And the
[33:42] governance from the start our design decision was the governance should be inherited and we don't rebuild it per agent. Um so, as we were building agent architectures, what we realized is that um obviously with agents accessing new
[33:58] data sets simultaneously, uh it could propose challenges and risks. So, our architecture needed to be watertight. Um we wanted governance to be inherited, not recreated with every agent that was being uh stood up. So, today Unity
[34:15] Catalog it manages like I think it is 184 tables and 5200 documented columns within this transportation mobility insights platform solution. And um
[34:35] yeah, I'm seeing if there's anything else I want to quickly cover. Um yeah, we had you know, it's it's definitely at scale. It's a Unity Catalog implementation at scale and it's a single governance layer for the entire platform. And concretely it was one plat one catalog organizing six
[34:51] schemas across the full life cycle bronze through gold and a dedicated TMI schema and a usage um analytics schema. So, I will just again highlight this that um the governance metadata, which is the
[35:06] classification, the ownership, the retention rules, the sharing approvals, all of that was assigned at the data asset level. So, wherever the data traveled, this was embedded into the metadata of it. Uh
[35:23] that meant that agents inherited those controls. Uh and again, we built the governance into the platform and not into each agent. So, and this became especially important as we started scaling from four use
[35:39] cases all the way to 17. So, a special mention on how we brought together the structured and the GIS data. I already covered that architecture in the other slide. So, uh in the interest of time, the only thing
[35:55] I will add here is that um yeah, the benefit for Caltrans was was twofold. So, one we talked about that with this architecture, you really didn't need a GIS analyst to do spatial analytics um because of all the components uh
[36:11] within which allowed you to do a natural language query. The second I think is very like transportation specific is that every output that we got was um us aligned to a Caltrans postmile, uh not a generic latitude longitude. And
[36:28] this is very important for Caltrans, which means that whatever insights they were trying to get, it aligns to a traffic management plan. Um, and uh, which is already a workflow that the, you know, that the agency uses, rather than needing a translation
[36:44] of the output of this tool. Uh, and then because this design is modular, a future routing or detour engine just fits without re-architecting all of it. And I will cover my last slide and then Duper will uh, talk about the outcomes.
[37:00] So, our vision is actually fairly simple. We want to uh, our long-term vision is to continuously um, consolidate the platform. Um, so today um, so really our our our
[37:20] Today there are some models, application, and agents which are outside of the core Databricks environment. And in the future state, we want to bring model serving, MLops, agent orchestration all on the single platform of Databricks. Unity catalog, I mean the governance is already on Databricks. The data is already in
[37:37] within the lakehouse. Um, and very specifically across the three tracks, like within model serving, there's a particular model, the DCR um, and then speed prediction model. These endpoints are outside of the platform. We want to bring this within the
[37:53] platform. Um, and the vision is that the full MLops life cycle operates end-to-end within uh, within the Databricks platform. And on the codebase, today there are a few cloud-native services and custom code that are still outside
[38:08] of Databricks. Um, we intend to repackage them into Databricks application. Um, and the vision is one one runtime and one CICD path. And um, last but not the least on the agents, today they they are distributed across two platforms. So, there is a
[38:25] possibility of fragmented orchestration, fragmented governance. Again, our intent is to bring all of them onto Databricks and really have they have Databricks be like a single source of truth for agent behavior, prompts, and evaluation.
[38:40] So, we talked a lot about architecture. We talked a lot about like the how it looks, but none of this is impactful unless there are outcomes. And Duper will now cover some of the exciting outcomes we saw from the
[38:56] solution. So, is my mic on? Okay. Okay, before I get the outcome, so I hope you guys learned something today. 20 minutes, we can talk hours, but I do learn a couple of things today. First, that's the first time I see the
[39:12] Databricks under hood. Now I Now I understand why my IT budget is so expensive to do this project. And second is I now I understand when she talked about the lake house, I thought she talked about her vacation time. Now I understand it's work related. So, the data lake house with
[39:29] the Databricks. Thank you so much for that. For the outcome, I'm not going to go for every every square on the slide. A few square I want to touch is really the the outcome is saving time. It's saving a lot of time, saving a lot of resource. Like
[39:45] I showed it like I I talked about the congestion report from thousand hours to second. That's amazing. And she touched a little about scalable. When we start the project, it take a months. I I was I was the first one to say I
[40:02] don't believe it. When Accenture told me, "Oh, we'll give you six months." I was like IT project six months? No way. Six years, right? They No, they said we'll six months. So, I see that. And actually we are looking for how to scale it up to do more use case. We are not stopping.
[40:20] We're investing. We are we want to expand the the technology and expand the use case to do more and more efficient. And then the data. The data is used to be take hours, now take minutes to take inside. Plus
[40:36] we find out we can get more insight from what we have. What I mean is we used to like to collect more data and more data and more data. But we find out oh, we do have a lot of data from other division people. So we can share, leverage the resource without spending
[40:52] more. Actually, we can do more. That's the efficiency I mentioned. And last I want to touch is how to make sure have a successful uh AI adoption and how to address the AI resistant. I mean you hear that people scared about the AI. Well, even we are
[41:10] at the State Department, we cannot make people just do it. We still need to work with the staff. Couple of thing I want to share is you may you may not believe at first when we approach this GenAI. We don't treat as a IT project.
[41:27] It's not a IT project. We are not as IT company hey give me a tools let me use it. We invest. This is a business project. The business people need a room. So what we do is we work closely with
[41:42] the Accenture, with the developer sitting in one room with the subject matter expert, with our local engineer to look at what kind of what kind of workflow. Kind of like you have the workflow agents sitting there to break down the workflow how we can use this
[41:58] technology to find efficiency. What is possibility? What's the option we get? And also we need the early executive team buying from our director, our chief deputy to invest the funding. That's why it's so expensive. They need to make decision in master funding to to use the
[42:15] technology even it's unknown at that time. And lastly is we conduct a lot of workshop. We train hundreds and hundreds of rank and file engineer. Uh in the beginning we do a post and before and after training. Beginning
[42:32] they say they don't know what's AI. And they don't know Caltrain do have AI application. But after the training the result is first 90% say I don't I don't know AI. I
[42:48] don't know Caltrain AI. After the training is 90% of people said I like what you're doing. I want more. More than what you have today. And 90% of the people feedback is I'm going to use it in my daily work.
[43:05] So that's a how you can address those AI adoption and how to address the AI resistant. Turning back to you. Yep. Any question I think the audience can ask. Yeah. We are we are over time but I will just add just one thing. I
[43:20] definitely we can talk about how brilliant the architecture is forever but none of that was so so we intentionally so deployed an MVP in August of last year and we had just 80
[43:35] users in the system then and that's when like business plan the series of adoption sessions, right? And it's it's just by district multiple adoption sessions per district. So if you're anybody who's implementing the solution, I would say like adoption is
[43:51] at the heart of it. However good your solution is, however smart your solution is, it's immaterial unless you are you know doing the adoption with your end users. And our end users are traffic engineers in the district. So we had to be and planners and you know a few other safety engineers. So, we had to make our
[44:08] adoption sessions like we had to speak their language. So, it was definitely a very collaborative effort between Accenture and and Dupa and his team to do that. So, that's that's the only thing I'll add. This concludes our presentation and we went 5 minutes over. So,
[44:25] we can be down here if there are any questions and then you have our email IDs Ajit Singh at Accenture Dupa Tong is at Capital One Financial and then Rex Phillips at Accenture, too. So, feel free to email us if you need anything. Thank you.
[44:41] Thank you, sweetie.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.