Amtrak's Digital Transformation with Databricks Lakehouse

Summary

Amtrak is undergoing its first major fleet renewal in 50 years — including 28 new Acela train sets between Boston and DC and 83 Siemens-built Arrow train sets for the West Coast — supported by a Databricks Lakehouse that unifies real-time train telemetry, reservation systems, infrastructure assets, and corporate data.
Lakeflow Connect simplifies data integration into a medallion architecture, and Unity Catalog provides governance, metadata, and lineage across all data products supporting fleet intelligence, predictive monitoring, and a $50 billion capital investment pipeline.
A planned Databricks Apps experience layer will embed Genie capabilities to give developers, analysts, and executives a shared self-service interface for discovering and analyzing trusted data across Amtrak's enterprise.

Amtrak's Digital Transformation with Databricks Lakehouse

Amtrak's digital transformation with Databricks Lakehouse connects real-time train telemetry, infrastructure assets, reservation systems, and corporate data. This unified foundation supports fleet intelligence, predictive monitoring, operational readiness, and planning for a $50 billion capital investment pipeline.
Learn how Amtrak uses Lakeflow Connect to simplify data integration, medallion architecture to curate data, and Unity Catalog to provide governance, metadata, and lineage. The session also covers analytical data products for fleet health, maintenance windows, reservations, and capital prioritization. Amtrak's planned Databricks Apps experience layer embeds Genie capabilities to help developers, analysts, and executives discover and analyze trusted data through a shared, self-service interface.
Data-driven fleet management: https://community.databricks.com/t5/databricks-free-edition-help/best-approaches-to-build-a-data-driven-fleet-management-system/td-p/149902
Genie Code and Lakeflow: https://www.databricks.com/blog/agentic-data-engineering-genie-code-and-lakeflow

Chapters

FAQs

What physical transformation is Amtrak currently undergoing?

Amtrak is acquiring new Acela train sets running between Boston and DC and 83 Siemens-built Arrow train sets for the West Coast, with a bid under way for new long-distance fleet as well. This is described as the first significant fleet renewal in the last 50 years.

How does Amtrak use the Databricks Data and AI platform for fleet intelligence?

Amtrak connects real-time train telemetry, infrastructure asset data, reservation systems, and corporate data through a Databricks Lakehouse with medallion architecture. This unified foundation supports fleet health monitoring, predictive maintenance, and operational readiness data products.

What is Lakeflow Connect and how does Amtrak use it?

Lakeflow Connect is a Databricks capability that simplifies data integration from multiple source systems into the lakehouse. Amtrak uses it to ingest data from disparate systems including train telemetry, reservations, and infrastructure assets, reducing the complexity of building and maintaining data pipelines.

How does Amtrak plan to make data accessible to non-technical business users?

Amtrak is building a Databricks Apps experience layer that embeds Genie capabilities so that developers, analysts, and executives can discover and analyze trusted data through a shared self-service interface. This planned layer aims to make the governed lakehouse accessible without requiring SQL expertise.

Full transcript

[01:07] Please help me welcome Pretheeba from Amtrak TO THE STAGE. I DON'T KNOW HOW MANY OF YOU KIND OF UH KNOW that Amtrak is about five decades old.
[01:23] Back from the Gilded Age, private railroads. I learned this history when I joined Amtrak about two years ago. is when the NRPC uh Amtrak was formed as a national
[01:38] Can you hear me now? Okay, I'm going to I'm not yelling, but I'm trying to reach the back of the room. Um Amtrak was formed in 1971 after a bunch of private railroads,
[01:55] and then the airline industry took off. We built the interstates and, you know, auto lobbies and whatnot. Um, to create the national passenger, the only passenger railroad in the US. We have one more.
[02:11] Uh, okay, one more passenger railroad in Florida. I'll not name them. Um, but this is the the largest, uh, network. We have about 21,000 track miles.
[02:26] 97% of those track miles are not owned by Amtrak. It's all running on freight railroad tracks, what we call as host railroads. Um, this is a significant moment for Amtrak and this is kind of what got me into Amtrak, um,
[02:42] uh, the last in the last couple of years. Um, Amtrak is going through a massive physical transformation. Everything that you saw in that video, it's not just a concept, it's happening. It's real. Um, there's, uh, not only investment in infrastructure,
[02:58] there is new fleet. Acela, how many of you have ridden an Acela? Um, we have new Acela fleet now running between Boston and DC. Uh, 28 train sets. And for those of you here on the West
[03:14] Coast, especially, you know, if you're in Portland, Seattle area, coming this fall is our Arrow train sets, uh, built by Siemens. We are acquiring 83 of those train sets. Um, and then very recently,
[03:29] there is an, um, a bit to kind of go, uh, get new long-distance fleet as well, right? Kind of long-distance travel, think about New York to Chicago, Chicago to LA, those types of lines. So, there is significant physical
[03:46] transformation that's happening at Amtrak. And, you know, this is the first of its kind in the last 50 years. Um and not only are we doing physical transformation, we are in the midst of one of the biggest digital transformations in the organization.
[04:02] Um which is how 15 years uh doing data and intelligence uh initiatives at financial services companies. And uh when Amtrak kind of uh got my attention a couple of years ago.
[04:19] Uh kind of what I believe the the possibilities here um to be able to actually be part of this massive transformation. This is an important moment uh for passenger railroad here in the US. So, the hardware, right? Is where the
[04:36] hard We have a lot of uh a fleet rolling stock about 20 units. 500 of them are um what we call as locomotives, the you know, the power cars. And about 2,000 of them are um
[04:54] the pas um Each one of them, including our new train sets, are data generating assets. 100 plus sensors in each one of these trains. And the modern train and the Aero train sets
[05:10] um have sensor mesh that send send real telemetry to our back office. Uh our mechanical teams are not reacting to uh issues, but they get predictive signals, right? Monitoring.
[05:25] Um these train sets are equipped with, you know, things like uh refrigerated refrig refrigerator sensors, right? So, that food safety checks. So, on so on the real-time capture. And for to process this data, we needed
[05:42] a platform such as Databricks, right? And to be able to combine that with the rest of our to be able to kind of make it available for our uh mechanical teams, safety teams, um infrastructure organizations.
[06:02] Collecting data of our trains, it's not just the train. We are also the efforts going on on infrastructure perspective. Um Then 50 billion dollars in investment uh in capital projects.
[06:18] These This pipeline, the 50 billion dollar pipeline, needs it's data intensive. We need the data to plan, build, maintain, and prioritize these efforts properly. These are long-running projects. These are not like 3-month, 6-month efforts.
[06:33] In order to kind of keep pace with all of these investments, it's data intensive. We need to be able to kind of capture that and make it available to our capital planning organization. So, and we are doing this with Lakehouse. So, I'm going to quickly touch upon like
[06:50] what the architecture looks like. I'm not going to drain this slide a whole lot. It's It's a very simplified architecture uh for all the intents and purposes here. Um Our Lakehouse platform selection was a strategic one, right? This is what is
[07:07] going to help us scale and grow as we are going through this massive physical transformation, right? So, we need the ability to bring data from all of our key sources, whether that's the fleet, the infrastructure assets, um whether that's kind of our corporate
[07:24] systems, reservation systems, um marketing, engagement data, bring it all together. Lake Flow Connect is part of our kind of the integration pipeline, simplifying the integration. We are taking advantage of all of the Lake Flow Connect capabilities, and we will I
[07:41] can't spot Rory here, but his commitment to support the space, we're going to keep taking advantage and pushing the product teams here at Databricks to kind of enable even more capabilities for us, right? What we have taken advantage of is still probably, uh you know, we're scratching
[07:58] the surface here. Uh but still there are some nuanced and unique challenges, uh especially in our real-time space, where we'll continue to partner with Databricks to kind of take advantage of it. So, we're leveraging the medallion architecture, kind of and kind of uh
[08:15] enriching the data, curating the data, make it making it consumable. And we are using Unity to kind of govern the data, um and making advanced analytics possible.
[08:32] So, the a glimpse of the use cases or how we think about that as data and analytical products that we are building through the lakehouse platform. The first uh that we have launched is around the fleet intelligence, which is around our uh S L N the arrow data sets, um also
[08:47] bringing in some of our legacy fleet health information and making it available uh for the various uh organizations. But we are continuing to build a significant pipeline of those data and analytical products within the platform, whether that's for operational readiness to kind of plan and, you know,
[09:04] schedule the work around our fleet, crew, consists, maintenance windows, or whether we're bringing in and modernizing our reservation uh platform and bringing the booking and the reservation data and providing data products and analytics analytical products supporting our commercial
[09:19] organization, or to help with our capital prioritization um processes. Um and I am running a little little short on time here. So, we have taken advantage of the platform to make it
[09:35] connect the data sources that we need to bring together. Now we have the governance framework around it. We're building the observability into the platform, but we want to continue taking advantage of the predictive and the compounding capability the platform has to offer, right? So, including taking
[09:50] advantage of the genie capabilities and all the announcements that came about in the last couple of days, right? But, I want to This is kind of the holy grail of any data and analytics journey. Um It's hard to touch and feel
[10:07] data, right? In an organization. The only way people have been used to consuming data is dashboards. How many of you still have dashboards? Reports, dashboards.
[10:24] How many of you have gotten rid of reports and dashboards? Not a single one of them. I don't think so we're going to get rid of them. Excel, macros. How many of them? Love it. Yeah, we all do, right? Um so, we are trying to create an experience layer that makes
[10:41] data and intelligence more consumable and something that you can touch and feel and make it less abstract for our consumer. This is part of our mission to make it self-service, to democratize data and intelligence across Amtrak. And
[10:56] here is a little bit of a concept that we are kind of trying to embark on. We see there's no audio to this one, so this is a Databricks apps experience a concept,
[11:13] which will pull all of the data and the analytical products into this experience layer. Helps you navigate and look at all the the metadata that's coming out of Unity, as well as be able to see the lineage
[11:28] in here Uh, the data products that we have created and it is one-stop shop, right? Whether you're a developer, uh, whether you're an analyst, uh, whether you're an executive, you have kind of one place, one experience
[11:44] layer to come in to be able to kind of shop for your data, analyze, discover, and to be able to work with this. So, there is, uh, a piece of, uh, the the Genie engine that's embedded embedded into this experience,
[12:00] uh, layer as well. And we're going to continue to kind of build this, uh, vision out for Amtrak and to kind of democratize data across Amtrak.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.